2024-07-02T22:25:20.956539Z info FLAG: --concurrency="0" 2024-07-02T22:25:20.956578Z info FLAG: --domain="echo1-4-84959.svc.cluster.local" 2024-07-02T22:25:20.956591Z info FLAG: --help="false" 2024-07-02T22:25:20.956596Z info FLAG: --log_as_json="false" 2024-07-02T22:25:20.956601Z info FLAG: --log_caller="" 2024-07-02T22:25:20.956605Z info FLAG: --log_output_level="delta:debug" 2024-07-02T22:25:20.956609Z info FLAG: --log_rotate="" 2024-07-02T22:25:20.956614Z info FLAG: --log_rotate_max_age="30" 2024-07-02T22:25:20.956618Z info FLAG: --log_rotate_max_backups="1000" 2024-07-02T22:25:20.956623Z info FLAG: --log_rotate_max_size="104857600" 2024-07-02T22:25:20.956627Z info FLAG: --log_stacktrace_level="default:none" 2024-07-02T22:25:20.956654Z info FLAG: --log_target="[stdout]" 2024-07-02T22:25:20.956660Z info FLAG: --meshConfig="./etc/istio/config/mesh" 2024-07-02T22:25:20.956664Z info FLAG: --outlierLogPath="" 2024-07-02T22:25:20.956668Z info FLAG: --profiling="true" 2024-07-02T22:25:20.956673Z info FLAG: --proxyComponentLogLevel="misc:error" 2024-07-02T22:25:20.956677Z info FLAG: --proxyLogLevel="warning" 2024-07-02T22:25:20.956681Z info FLAG: --serviceCluster="istio-proxy" 2024-07-02T22:25:20.956686Z info FLAG: --stsPort="0" 2024-07-02T22:25:20.956690Z info FLAG: --templateFile="" 2024-07-02T22:25:20.956694Z info FLAG: --tokenManagerPlugin="" 2024-07-02T22:25:20.956707Z info FLAG: --vklog="0" 2024-07-02T22:25:20.956713Z info Version 1.23-dev-bac055d1e045b65edba399c2d215d9ecf4e21ad8-Clean 2024-07-02T22:25:20.956723Z info Set max file descriptors (ulimit -n) to: 1048576 2024-07-02T22:25:20.957044Z info Proxy role ips=[10.20.0.31] type=sidecar id=migration-vistio-df6bc8969-rsrmr.echo1-4-84959 domain=echo1-4-84959.svc.cluster.local 2024-07-02T22:25:20.957141Z info Apply proxy config from env {"proxyMetadata":{"ISTIO_META_DNS_CAPTURE":"true"},"gatewayTopology":{"numTrustedProxies":1},"meshId":"testmesh0"} 2024-07-02T22:25:20.963350Z info cpu limit detected as 2, setting concurrency 2024-07-02T22:25:20.974875Z info Effective config: binaryPath: /usr/local/bin/envoy concurrency: 2 configPath: ./etc/istio/proxy controlPlaneAuthPolicy: MUTUAL_TLS discoveryAddress: istiod.istio-system.svc:15012 drainDuration: 45s gatewayTopology: numTrustedProxies: 1 meshId: testmesh0 proxyAdminPort: 15000 proxyMetadata: ISTIO_META_DNS_CAPTURE: "true" serviceCluster: istio-proxy statNameLength: 189 statusPort: 15020 terminationDrainDuration: 5s 2024-07-02T22:25:20.974952Z info JWT policy is third-party-jwt 2024-07-02T22:25:20.974962Z info using credential fetcher of JWT type in cluster.local trust domain 2024-07-02T22:25:20.983138Z info platform detected is GCP 2024-07-02T22:25:20.987971Z info Prometheus scraping configuration: {true 15014} 2024-07-02T22:25:20.989424Z info dns Starting local udp DNS server on 127.0.0.1:15053 2024-07-02T22:25:20.989672Z info dns Starting local tcp DNS server on 127.0.0.1:15053 2024-07-02T22:25:20.989752Z info Workload SDS socket not found. Starting Istio SDS Server 2024-07-02T22:25:20.989772Z info CA Endpoint istiod.istio-system.svc:15012, provider Citadel 2024-07-02T22:25:20.989800Z info Using CA istiod.istio-system.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem 2024-07-02T22:25:20.990123Z info Opening status port 15020 2024-07-02T22:25:21.001783Z info xdsproxy Initializing with upstream address "istiod.istio-system.svc:15012" and cluster "remote" 2024-07-02T22:25:21.026894Z info sds Starting SDS grpc server 2024-07-02T22:25:21.027058Z info sds Starting SDS server for workload certificates, will listen on "./var/run/secrets/workload-spiffe-uds/socket" 2024-07-02T22:25:21.027360Z info starting Http service at 127.0.0.1:15004 2024-07-02T22:25:21.090793Z info Pilot SAN: [istiod.istio-system.svc] 2024-07-02T22:25:21.116784Z info Starting proxy agent 2024-07-02T22:25:21.116859Z info Envoy command: [-c etc/istio/proxy/envoy-rev.json --drain-time-s 45 --drain-strategy immediate --local-address-ip-version v4 --file-flush-interval-msec 1000 --disable-hot-restart --allow-unknown-static-fields -l warning --component-log-level misc:error --concurrency 2] 2024-07-02T22:25:22.827192Z info cache generated new workload certificate latency=1.810276306s ttl=23h59m59.172821096s 2024-07-02T22:25:22.827349Z info cache Root cert has changed, start rotating root cert 2024-07-02T22:25:22.827508Z info cache returned workload trust anchor from cache ttl=23h59m59.172495039s 2024-07-02T22:25:25.206082Z warning envoy main external/envoy/source/server/server.cc:843 Usage of the deprecated runtime key overload.global_downstream_max_connections, consider switching to `envoy.resource_monitors.downstream_connections` instead.This runtime key will be removed in future. thread=19 2024-07-02T22:25:25.219954Z warning envoy main external/envoy/source/server/server.cc:936 There is no configured limit to the number of allowed active downstream connections. Configure a limit in `envoy.resource_monitors.downstream_connections` resource monitor. thread=19 2024-07-02T22:25:25.386562Z info xdsproxy connected to delta upstream XDS server: istiod.istio-system.svc:15012 id=1 2024-07-02T22:25:33.345747Z info ads ADS: new connection for node:1 2024-07-02T22:25:33.346352Z info cache returned workload certificate from cache ttl=23h59m48.653655434s 2024-07-02T22:25:33.365608Z info ads ADS: new connection for node:2 2024-07-02T22:25:33.366028Z info cache returned workload trust anchor from cache ttl=23h59m48.6339776s 2024-07-02T22:25:48.370790Z info Readiness succeeded in 27.415874382s 2024-07-02T22:25:48.371861Z info Envoy proxy is ready 2024-07-02T22:26:32.962000Z warn xdsproxy sendDownstreamDelta took 38.585466823s 2024-07-02T22:26:32.962152Z warn xdsproxy downstream terminated with unexpected error send error for type url type.googleapis.com/envoy.config.route.v3.RouteConfiguration: rpc error: code = Unavailable desc = transport is closing id=1 2024-07-02T22:26:33.013814Z error xdsproxy downstream dropped delta xds push to Envoy, connection already closed id=1 2024-07-02T22:26:33.014413Z error xdsproxy downstream dropped delta xds push to Envoy, connection already closed id=1 2024-07-02T22:27:11.661316Z warning envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:176 DeltaAggregatedResources gRPC config stream to xds-grpc closed: 13, thread=19 2024-07-02T22:27:16.538570Z info xdsproxy connected to delta upstream XDS server: istiod.istio-system.svc:15012 id=2 2024-07-02T22:29:25.068014Z info xdsproxy connected to delta upstream XDS server: istiod.istio-system.svc:15012 id=3 2024-07-02T22:30:06.603247Z warn xdsproxy sendDownstreamDelta took 12.642132774s 2024-07-02T22:30:06.603543Z warn xdsproxy downstream terminated with unexpected error send error for type url type.googleapis.com/envoy.config.route.v3.RouteConfiguration: rpc error: code = Unavailable desc = transport is closing id=3 2024-07-02T22:30:06.604083Z error xdsproxy downstream dropped delta xds push to Envoy, connection already closed id=3 2024-07-02T22:30:20.094246Z warning envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:176 DeltaAggregatedResources gRPC config stream to xds-grpc closed: 13, thread=19 2024-07-02T22:30:22.374397Z info xdsproxy connected to delta upstream XDS server: istiod.istio-system.svc:15012 id=4 2024-07-02T22:31:08.964487Z info ads ADS: "" 1 terminated with stream closed 2024-07-02T22:31:08.964540Z info ads ADS: "" 2 terminated 2024-07-02T22:32:20.535908Z info ads ADS: new connection for node:3 2024-07-02T22:32:20.536185Z info cache returned workload certificate from cache ttl=23h53m1.463820782s 2024-07-02T22:32:20.535936Z info ads ADS: new connection for node:4 2024-07-02T22:32:20.536659Z info cache returned workload trust anchor from cache ttl=23h53m1.463344657s 2024-07-02T22:32:26.159233Z info xdsproxy connected to delta upstream XDS server: istiod.istio-system.svc:15012 id=5 2024-07-02T22:33:41.057733Z warn xdsproxy sendDownstreamDelta took 35.701703019s 2024-07-02T22:33:41.057831Z warn xdsproxy downstream terminated with unexpected error send error for type url type.googleapis.com/envoy.config.route.v3.RouteConfiguration: rpc error: code = Unavailable desc = transport is closing id=5 2024-07-02T22:34:22.833105Z warning envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:176 DeltaAggregatedResources gRPC config stream to xds-grpc closed: 13, thread=19 2024-07-02T22:34:24.428944Z info xdsproxy connected to delta upstream XDS server: istiod.istio-system.svc:15012 id=6 2024-07-02T22:35:44.640367Z warning envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:176 DeltaAggregatedResources gRPC config stream to xds-grpc closed: 13, thread=19 2024-07-02T22:35:51.968127Z info xdsproxy connected to delta upstream XDS server: istiod.istio-system.svc:15012 id=7 2024-07-02T22:36:38.754643Z info ads ADS: "" 3 terminated 2024-07-02T22:36:38.754744Z info ads ADS: "" 4 terminated 2024-07-02T22:37:17.831269Z warn xdsproxy sendDownstreamDelta took 25.830481204s 2024-07-02T22:37:17.831361Z warn xdsproxy downstream terminated with unexpected error send error for type url type.googleapis.com/envoy.config.route.v3.RouteConfiguration: rpc error: code = Unavailable desc = transport is closing id=7 2024-07-02T22:37:17.832139Z error xdsproxy downstream dropped delta xds push to Envoy, connection already closed id=7 2024-07-02T22:37:23.040673Z warning envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:176 DeltaAggregatedResources gRPC config stream to xds-grpc closed: 13, thread=19 2024-07-02T22:37:23.044508Z info ads ADS: new connection for node:5 2024-07-02T22:37:23.044668Z info cache returned workload trust anchor from cache ttl=23h47m58.955336271s 2024-07-02T22:37:23.045343Z info ads ADS: new connection for node:6 2024-07-02T22:37:23.045609Z info cache returned workload certificate from cache ttl=23h47m58.954396079s 2024-07-02T22:37:31.146795Z critical envoy assert external/envoy/source/extensions/config_subscription/grpc/delta_subscription_state.cc:344 assert failure: !requested_resource_state_.empty(). thread=19 2024-07-02T22:37:31.147056Z critical envoy backtrace external/envoy/source/server/backtrace.h:127 Caught Aborted, suspect faulting address 0x53900000013 thread=19 2024-07-02T22:37:31.147132Z critical envoy backtrace external/envoy/source/server/backtrace.h:111 Backtrace (use tools/stack_decode.py to get line numbers): thread=19 2024-07-02T22:37:31.147203Z critical envoy backtrace external/envoy/source/server/backtrace.h:112 Envoy version: 1fd1ca76e8e66e7d3cc68ae7eac6bed681d6d8da/1.31.0-dev/Clean/DEBUG/BoringSSL thread=19 2024-07-02T22:37:31.147257Z critical envoy backtrace external/envoy/source/server/backtrace.h:114 Address mapping: 5592dfdc4000-5592e9543000 /usr/local/bin/envoy thread=19 2024-07-02T22:37:31.391005Z critical envoy backtrace external/envoy/source/server/backtrace.h:119 #0: Envoy::SignalAction::sigHandler() [0x5592e7db9e9c] thread=19 2024-07-02T22:37:31.391929Z critical envoy backtrace external/envoy/source/server/backtrace.h:121 #1: [0x7c27d4588320] thread=19 2024-07-02T22:37:31.392647Z critical envoy backtrace external/envoy/source/server/backtrace.h:119 #2: raise [0x7c27d458826e] thread=19 2024-07-02T22:37:31.393233Z critical envoy backtrace external/envoy/source/server/backtrace.h:119 #3: abort [0x7c27d456b8ff] thread=19 2024-07-02T22:37:31.636208Z critical envoy backtrace external/envoy/source/server/backtrace.h:119 #4: Envoy::Config::DeltaSubscriptionState::isInitialRequestForLegacyWildcard() [0x5592e547f73f] thread=19 2024-07-02T22:37:31.885716Z critical envoy backtrace external/envoy/source/server/backtrace.h:119 #5: Envoy::Config::DeltaSubscriptionState::getNextRequestAckless() [0x5592e547e369] thread=19 2024-07-02T22:37:32.162486Z critical envoy backtrace external/envoy/source/server/backtrace.h:119 #6: Envoy::Config::NewGrpcMuxImpl::trySendDiscoveryRequests() [0x5592e5452021] thread=19 2024-07-02T22:37:33.451145Z critical envoy backtrace external/envoy/source/server/backtrace.h:119 #7: Envoy::Config::NewGrpcMuxImpl::onStreamEstablished() [0x5592e545328b] thread=19 2024-07-02T22:37:35.031405Z critical envoy backtrace external/envoy/source/server/backtrace.h:119 #8: Envoy::Config::GrpcStream<>::establishNewStream() [0x5592e5463a88] thread=19 2024-07-02T22:37:37.664150Z critical envoy backtrace external/envoy/source/server/backtrace.h:119 #9: Envoy::Config::GrpcStream<>::GrpcStream()::{lambda()#1}::operator()() [0x5592e54657b9] thread=19 2024-07-02T22:37:38.229391Z critical envoy backtrace external/envoy/source/server/backtrace.h:119 #10: _ZNSt3__18__invokeIRZN5Envoy6Config10GrpcStreamIN5envoy7service9discovery2v321DeltaDiscoveryRequestENS7_22DeltaDiscoveryResponseEEC1EPNS2_19GrpcStreamCallbacksIS9_EENS_10unique_ptrINS1_4Grpc14RawAsyncClientENS_14default_deleteISG_EEEERKN6google8protobuf16MethodDescriptorERNS1_5Event10DispatcherERNS1_5Stats5ScopeENSE_INS1_15BackOffStrategyENSH_ISV_EEEERKNS2_17RateLimitSettingsEEUlvE_JEEEDTclscT_fp_spscT0_fp0_EEOS13_DpOS14_ [0x5592e5465785] thread=19 2024-07-02T22:37:38.905063Z critical envoy backtrace external/envoy/source/server/backtrace.h:119 #11: std::__1::__invoke_void_return_wrapper<>::__call<>() [0x5592e546573d] thread=19 2024-07-02T22:37:40.144565Z critical envoy backtrace external/envoy/source/server/backtrace.h:119 #12: std::__1::__function::__alloc_func<>::operator()() [0x5592e546570d] thread=19 2024-07-02T22:37:40.933133Z critical envoy backtrace external/envoy/source/server/backtrace.h:119 #13: std::__1::__function::__func<>::operator()() [0x5592e5464879] thread=19 2024-07-02T22:37:41.419589Z critical envoy backtrace external/envoy/source/server/backtrace.h:119 #14: std::__1::__function::__value_func<>::operator()() [0x5592dff07742] thread=19 2024-07-02T22:37:41.741848Z critical envoy backtrace external/envoy/source/server/backtrace.h:119 #15: std::__1::function<>::operator()() [0x5592dff06fb5] thread=19 2024-07-02T22:37:42.116164Z critical envoy backtrace external/envoy/source/server/backtrace.h:119 #16: Envoy::Event::DispatcherImpl::createTimerInternal()::$_20::operator()() [0x5592e78e65c9] thread=19 2024-07-02T22:37:42.291022Z critical envoy backtrace external/envoy/source/server/backtrace.h:119 #17: _ZNSt3__18__invokeIRZN5Envoy5Event14DispatcherImpl19createTimerInternalENS_8functionIFvvEEEE4$_20JEEEDTclscT_fp_spscT0_fp0_EEOS9_DpOSA_ [0x5592e78e6585] thread=19 2024-07-02T22:37:42.517465Z critical envoy backtrace external/envoy/source/server/backtrace.h:119 #18: std::__1::__invoke_void_return_wrapper<>::__call<>() [0x5592e78e653d] thread=19 2024-07-02T22:37:42.769227Z critical envoy backtrace external/envoy/source/server/backtrace.h:119 #19: std::__1::__function::__alloc_func<>::operator()() [0x5592e78e650d] thread=19 2024-07-02T22:37:43.400619Z critical envoy backtrace external/envoy/source/server/backtrace.h:119 #20: std::__1::__function::__func<>::operator()() [0x5592e78e5719] thread=19 2024-07-02T22:37:43.400690Z critical envoy backtrace external/envoy/source/server/backtrace.h:119 #21: std::__1::__function::__value_func<>::operator()() [0x5592dff07742] thread=19 2024-07-02T22:37:43.400709Z critical envoy backtrace external/envoy/source/server/backtrace.h:119 #22: std::__1::function<>::operator()() [0x5592dff06fb5] thread=19 2024-07-02T22:37:44.460544Z critical envoy backtrace external/envoy/source/server/backtrace.h:119 #23: Envoy::Event::TimerImpl::TimerImpl()::$_1::operator()() [0x5592e7db8e3c] thread=19 2024-07-02T22:37:45.042166Z critical envoy backtrace external/envoy/source/server/backtrace.h:119 #24: Envoy::Event::TimerImpl::TimerImpl()::$_1::__invoke() [0x5592e7db8de9] thread=19 2024-07-02T22:37:45.393666Z critical envoy backtrace external/envoy/source/server/backtrace.h:119 #25: event_process_active_single_queue [0x5592e81a928c] thread=19 2024-07-02T22:37:47.599880Z critical envoy backtrace external/envoy/source/server/backtrace.h:119 #26: event_process_active [0x5592e81a3ae8] thread=19 2024-07-02T22:37:48.969413Z critical envoy backtrace external/envoy/source/server/backtrace.h:119 #27: event_base_loop [0x5592e81a29ec] thread=19 2024-07-02T22:37:49.209691Z critical envoy backtrace external/envoy/source/server/backtrace.h:119 #28: Envoy::Event::LibeventScheduler::run() [0x5592e7db5c5f] thread=19 2024-07-02T22:37:49.403510Z critical envoy backtrace external/envoy/source/server/backtrace.h:119 #29: Envoy::Event::DispatcherImpl::run() [0x5592e78d9159] thread=19 2024-07-02T22:37:49.621236Z critical envoy backtrace external/envoy/source/server/backtrace.h:119 #30: Envoy::Server::InstanceBase::run() [0x5592e5fa6e24] thread=19 2024-07-02T22:37:49.975688Z critical envoy backtrace external/envoy/source/server/backtrace.h:119 #31: Envoy::StrippedMainBase::runServer() [0x5592e5e54594] thread=19 2024-07-02T22:37:50.391970Z critical envoy backtrace external/envoy/source/server/backtrace.h:119 #32: Envoy::MainCommonBase::run() [0x5592e5e4f338] thread=19 2024-07-02T22:37:50.803684Z critical envoy backtrace external/envoy/source/server/backtrace.h:119 #33: Envoy::MainCommon::run() [0x5592e5e54f6c] thread=19 2024-07-02T22:37:51.264278Z critical envoy backtrace external/envoy/source/server/backtrace.h:119 #34: Envoy::MainCommon::main() [0x5592e5e505bc] thread=19 2024-07-02T22:37:51.747709Z critical envoy backtrace external/envoy/source/server/backtrace.h:119 #35: main [0x5592dff71e45] thread=19 2024-07-02T22:37:51.748076Z critical envoy backtrace external/envoy/source/server/backtrace.h:121 #36: [0x7c27d456d1ca] thread=19 2024-07-02T22:37:51.748422Z critical envoy backtrace external/envoy/source/server/backtrace.h:119 #37: __libc_start_main [0x7c27d456d28b] thread=19 2024-07-02T22:37:52.091187Z critical envoy backtrace external/envoy/source/server/backtrace.h:119 #38: _start [0x5592dfdc402a] thread=19 2024-07-02T22:37:53.200785Z info ads ADS: "" 5 terminated 2024-07-02T22:37:53.207848Z info ads ADS: "" 6 terminated 2024-07-02T22:37:53.385701Z error Envoy exited with error: signal: aborted (core dumped)