INFO[2024-08-10T13:11:07Z] ci-operator version v20240809-280ef67b6      
INFO[2024-08-10T13:11:07Z] Loading configuration from https://config.ci.openshift.org for openshift/cluster-node-tuning-operator@master 
INFO[2024-08-10T13:11:07Z] Resolved source https://github.com/openshift/cluster-node-tuning-operator to master@6d625b5a, merging: #1129 e9952ee6 @ffromani 
WARN[2024-08-10T13:11:07Z] skipped directory "..2024_08_10_13_10_53.1280262370" when creating secret from directory "/secrets/ci-pull-credentials" 
WARN[2024-08-10T13:11:07Z] skipped directory "..2024_08_10_13_10_53.3180725782" when creating secret from directory "/usr/local/e2e-aws-operator-cluster-profile" 
INFO[2024-08-10T13:11:07Z] Loading information from https://config.ci.openshift.org for integrated stream ocp/4.18 
INFO[2024-08-10T13:11:07Z] Loading information from https://config.ci.openshift.org for integrated stream ocp/4.18 
INFO[2024-08-10T13:11:07Z] Building release initial from a snapshot of ocp/4.18 
INFO[2024-08-10T13:11:07Z] Building release latest from a snapshot of ocp/4.18 
INFO[2024-08-10T13:11:07Z] Using namespace https://console-openshift-console.apps.build03.ky4t.p1.openshiftapps.com/k8s/cluster/projects/ci-op-q69b2z86 
INFO[2024-08-10T13:11:07Z] Running [input:root], [input:ocp_4.17_base-rhel9], [input:ocp_builder_rhel-9-golang-1.22-openshift-4.17], [input:origin-centos-8], [input:ocp-4.12-upi-installer], [input:ocp-4.14-upi-installer], [input:ocp-4.16-upi-installer], [release-inputs:initial], [release-inputs:latest], src, cluster-node-tuning-operator, [output:stable:cluster-node-tuning-operator], [images], [release:latest], e2e-aws-operator 
INFO[2024-08-10T13:11:07Z] Tagging ocp/builder:rhel-9-golang-1.22-openshift-4.17 into pipeline:ocp_builder_rhel-9-golang-1.22-openshift-4.17. 
INFO[2024-08-10T13:11:07Z] Tagging ocp/4.14:upi-installer into pipeline:ocp-4.14-upi-installer. 
INFO[2024-08-10T13:11:07Z] Tagging ocp/4.16:upi-installer into pipeline:ocp-4.16-upi-installer. 
INFO[2024-08-10T13:11:07Z] Tagging ocp/4.12:upi-installer into pipeline:ocp-4.12-upi-installer. 
INFO[2024-08-10T13:11:07Z] Tagging openshift/release:rhel-9-release-golang-1.22-openshift-4.17 into pipeline:root. 
INFO[2024-08-10T13:11:07Z] Tagging origin/centos:8 into pipeline:origin-centos-8. 
INFO[2024-08-10T13:11:07Z] Tagging ocp/4.18:base-rhel9 into pipeline:ocp_4.17_base-rhel9. 
INFO[2024-08-10T13:11:07Z] Waiting to import tags on imagestream (after taking snapshot) ci-op-q69b2z86/stable-initial ... 
INFO[2024-08-10T13:11:07Z] Waiting to import tags on imagestream (after taking snapshot) ci-op-q69b2z86/stable ... 
INFO[2024-08-10T13:11:12Z] Building src                                 
INFO[2024-08-10T13:11:13Z] Found existing build "src-amd64"             
INFO[2024-08-10T13:12:59Z] Build src-amd64 succeeded after 1m26s        
INFO[2024-08-10T13:13:00Z] Image ci-op-q69b2z86/pipeline:src created     for-build=src
INFO[2024-08-10T13:13:00Z] Building cluster-node-tuning-operator        
INFO[2024-08-10T13:13:00Z] Found existing build "cluster-node-tuning-operator-amd64" 
INFO[2024-08-10T13:14:07Z] Imported tags on imagestream (after taking snapshot) ci-op-q69b2z86/stable 
INFO[2024-08-10T13:14:07Z] Imported tags on imagestream (after taking snapshot) ci-op-q69b2z86/stable-initial 
INFO[2024-08-10T13:16:49Z] Build cluster-node-tuning-operator-amd64 succeeded after 3m49s 
INFO[2024-08-10T13:16:50Z] Image ci-op-q69b2z86/pipeline:cluster-node-tuning-operator created  for-build=cluster-node-tuning-operator
INFO[2024-08-10T13:16:50Z] Tagging cluster-node-tuning-operator into stable 
INFO[2024-08-10T13:16:50Z] Creating release image registry.build03.ci.openshift.org/ci-op-q69b2z86/release:latest. 
INFO[2024-08-10T13:18:00Z] Snapshot integration stream into release 4.18.0-0.ci.test-2024-08-10-131650-ci-op-q69b2z86-latest to tag release:latest  
INFO[2024-08-10T13:18:00Z] Acquiring leases for test e2e-aws-operator: [aws-quota-slice] 
INFO[2024-08-10T13:18:00Z] Acquired 1 lease(s) for aws-quota-slice: [us-west-2--aws-quota-slice-30] 
INFO[2024-08-10T13:18:00Z] Acquiring IP Pool leases for test e2e-aws-operator: aws-ip-pools-us-west-2 
INFO[2024-08-10T13:18:00Z] no leases of type: aws-ip-pools-us-west-2 available 
INFO[2024-08-10T13:18:00Z] Running multi-stage test e2e-aws-operator    
INFO[2024-08-10T13:18:00Z] Running multi-stage phase pre                
INFO[2024-08-10T13:18:00Z] Running step e2e-aws-operator-ipi-conf.      
INFO[2024-08-10T13:18:11Z] Step e2e-aws-operator-ipi-conf succeeded after 10s. 
INFO[2024-08-10T13:18:11Z] Running step e2e-aws-operator-ipi-conf-telemetry. 
INFO[2024-08-10T13:18:18Z] Step e2e-aws-operator-ipi-conf-telemetry succeeded after 7s. 
INFO[2024-08-10T13:18:18Z] Running step e2e-aws-operator-ipi-conf-aws.  
INFO[2024-08-10T13:18:32Z] Step e2e-aws-operator-ipi-conf-aws succeeded after 13s. 
INFO[2024-08-10T13:18:32Z] Running step e2e-aws-operator-ipi-conf-aws-byo-ipv4-pool-public. 
INFO[2024-08-10T13:18:40Z] Step e2e-aws-operator-ipi-conf-aws-byo-ipv4-pool-public succeeded after 8s. 
INFO[2024-08-10T13:18:40Z] Running step e2e-aws-operator-ipi-install-monitoringpvc. 
INFO[2024-08-10T13:18:49Z] Step e2e-aws-operator-ipi-install-monitoringpvc succeeded after 8s. 
INFO[2024-08-10T13:18:49Z] Running step e2e-aws-operator-ipi-install-rbac. 
INFO[2024-08-10T13:18:57Z] Step e2e-aws-operator-ipi-install-rbac succeeded after 8s. 
INFO[2024-08-10T13:18:57Z] Running step e2e-aws-operator-openshift-cluster-bot-rbac. 
INFO[2024-08-10T13:19:06Z] Step e2e-aws-operator-openshift-cluster-bot-rbac succeeded after 8s. 
INFO[2024-08-10T13:19:06Z] Running step e2e-aws-operator-ipi-install-hosted-loki. 
INFO[2024-08-10T13:19:14Z] Step e2e-aws-operator-ipi-install-hosted-loki succeeded after 8s. 
INFO[2024-08-10T13:19:14Z] Running step e2e-aws-operator-ipi-install-install. 
INFO[2024-08-10T13:59:44Z] Step e2e-aws-operator-ipi-install-install succeeded after 40m30s. 
INFO[2024-08-10T13:59:44Z] Running step e2e-aws-operator-ipi-install-times-collection. 
INFO[2024-08-10T13:59:53Z] Step e2e-aws-operator-ipi-install-times-collection succeeded after 9s. 
INFO[2024-08-10T13:59:53Z] Running step e2e-aws-operator-nodes-readiness. 
INFO[2024-08-10T14:00:05Z] Step e2e-aws-operator-nodes-readiness succeeded after 11s. 
INFO[2024-08-10T14:00:05Z] Running step e2e-aws-operator-multiarch-validate-nodes. 
INFO[2024-08-10T14:00:13Z] Step e2e-aws-operator-multiarch-validate-nodes succeeded after 8s. 
INFO[2024-08-10T14:00:13Z] Step phase pre succeeded after 42m12s.       
INFO[2024-08-10T14:00:13Z] Running multi-stage phase test               
INFO[2024-08-10T14:00:13Z] Running step e2e-aws-operator-test.          
INFO[2024-08-10T14:07:36Z] Logs for container test in pod e2e-aws-operator-test: 
INFO[2024-08-10T14:07:36Z] GOOS=linux GO111MODULE=on GOFLAGS=-mod=vendor go build -o _output/go-bindata ./vendor/github.com/kevinburke/go-bindata/go-bindata
_output/go-bindata -mode 420 -modtime 1 -pkg manifests -o pkg/manifests/bindata.go assets/...
gofmt -s -w pkg/manifests/bindata.go
for d in core basic reboots reboots/sno deferred; do \
  KUBERNETES_CONFIG="/tmp/kubeconfig-3364950287" GOOS=linux GO111MODULE=on GOFLAGS=-mod=vendor go test -v -timeout 40m ./test/e2e/$d -ginkgo.v -ginkgo.no-color -ginkgo.fail-fast -ginkgo.label-filter=!flaky || exit; \
done
=== RUN   TestNodeTuningOperator
Running Suite: Node Tuning Operator e2e tests: core - /go/src/github.com/openshift/cluster-node-tuning-operator/test/e2e/core
=============================================================================================================================
Random Seed: 1723298458

Will run 1 of 1 specs
------------------------------
[core][cluster_version] Node Tuning Operator host, container OS and cluster version host, container OS and cluster version retrievable
/go/src/github.com/openshift/cluster-node-tuning-operator/test/e2e/core/cluster_version.go:20
  STEP: getting a list of worker nodes @ 08/10/24 14:00:58.762
  STEP: getting a TuneD Pod running on node ip-10-0-14-106.us-west-2.compute.internal @ 08/10/24 14:00:59.088
  STEP: getting the host OS version on node ip-10-0-14-106.us-west-2.compute.internal @ 08/10/24 14:00:59.157
  NAME="Red Hat Enterprise Linux CoreOS"
  ID="rhcos"
  ID_LIKE="rhel fedora"
  VERSION="416.94.202405210347-0"
  VERSION_ID="4.16"
  VARIANT="CoreOS"
  VARIANT_ID=coreos
  PLATFORM_ID="platform:el9"
  PRETTY_NAME="Red Hat Enterprise Linux CoreOS 416.94.202405210347-0"
  ANSI_COLOR="0;31"
  CPE_NAME="cpe:/o:redhat:enterprise_linux:9::baseos::coreos"
  HOME_URL="https://www.redhat.com/"
  DOCUMENTATION_URL="https://docs.okd.io/latest/welcome/index.html"
  BUG_REPORT_URL="https://access.redhat.com/labs/rhir/"
  REDHAT_BUGZILLA_PRODUCT="OpenShift Container Platform"
  REDHAT_BUGZILLA_PRODUCT_VERSION="4.16"
  REDHAT_SUPPORT_PRODUCT="OpenShift Container Platform"
  REDHAT_SUPPORT_PRODUCT_VERSION="4.16"
  OPENSHIFT_VERSION="4.16"
  RHEL_VERSION=9.4
  OSTREE_VERSION="416.94.202405210347-0"

  STEP: getting the TuneD container OS version @ 08/10/24 14:01:00.419
  NAME="Red Hat Enterprise Linux"
  VERSION="9.2 (Plow)"
  ID="rhel"
  ID_LIKE="fedora"
  VERSION_ID="9.2"
  PLATFORM_ID="platform:el9"
  PRETTY_NAME="Red Hat Enterprise Linux 9.2 (Plow)"
  ANSI_COLOR="0;31"
  LOGO="fedora-logo-icon"
  CPE_NAME="cpe:/o:redhat:enterprise_linux:9::baseos"
  HOME_URL="https://www.redhat.com/"
  DOCUMENTATION_URL="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9"
  BUG_REPORT_URL="https://issues.redhat.com/"

  REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 9"
  REDHAT_BUGZILLA_PRODUCT_VERSION=9.2
  REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
  REDHAT_SUPPORT_PRODUCT_VERSION="9.2"

  STEP: getting the cluster version @ 08/10/24 14:01:01.441
  run command 'oc [get clusterversion version -o jsonpath='{.status.desired.version}']':
    out='4.18.0-0.ci.test-2024-08-10-131650-ci-op-q69b2z86-latest'
    err=
    ret=<nil>
• [2.969 seconds]
------------------------------

Ran 1 of 1 Specs in 2.970 seconds
SUCCESS! -- 1 Passed | 0 Failed | 0 Pending | 0 Skipped
--- PASS: TestNodeTuningOperator (2.97s)
PASS
ok  	github.com/openshift/cluster-node-tuning-operator/test/e2e/core	2.987s
=== RUN   TestNodeTuningOperator
Running Suite: Node Tuning Operator e2e tests: basic - /go/src/github.com/openshift/cluster-node-tuning-operator/test/e2e/basic
===============================================================================================================================
Random Seed: 1723298463

Will run 15 of 15 specs
------------------------------
[basic][rollback] Node Tuning Operator settings rollback TuneD settings rollback net.ipv4.tcp_tw_reuse set
/go/src/github.com/openshift/cluster-node-tuning-operator/test/e2e/basic/rollback.go:54
  STEP: getting a list of worker nodes @ 08/10/24 14:01:03.662
  STEP: getting a TuneD Pod running on node ip-10-0-14-106.us-west-2.compute.internal @ 08/10/24 14:01:03.98
  STEP: waiting for TuneD profile openshift-node on node ip-10-0-14-106.us-west-2.compute.internal @ 08/10/24 14:01:04.048
  STEP: ensuring the default net.ipv4.tcp_tw_reuse value (2) is set in Pod tuned-785tw @ 08/10/24 14:01:04.113
  STEP: labelling Pod tuned-785tw with label tuned.openshift.io/ingress @ 08/10/24 14:01:05.13
  run command 'oc [label pod --overwrite -n openshift-cluster-node-tuning-operator tuned-785tw tuned.openshift.io/ingress=]':
    out=pod/tuned-785tw labeled
    err=
    ret=<nil>
  STEP: creating custom profile /go/src/github.com/openshift/cluster-node-tuning-operator/examples/ingress.yaml @ 08/10/24 14:01:05.547
  run command 'oc [create -n openshift-cluster-node-tuning-operator -f /go/src/github.com/openshift/cluster-node-tuning-operator/examples/ingress.yaml]':
    out=tuned.tuned.openshift.io/openshift-ingress created
    err=
    ret=<nil>
  STEP: ensuring the custom worker node profile was set @ 08/10/24 14:01:05.84
  STEP: deleting Pod tuned-785tw @ 08/10/24 14:01:06.853
  run command 'oc [delete -n openshift-cluster-node-tuning-operator pod tuned-785tw --wait]':
    out=pod "tuned-785tw" deleted
    err=
    ret=<nil>
  STEP: waiting for a new TuneD Pod to be ready on node ip-10-0-14-106.us-west-2.compute.internal @ 08/10/24 14:01:07.859
  STEP: ensuring the custom net.ipv4.tcp_tw_reuse value (1) is still set in Pod tuned-rn9gc @ 08/10/24 14:01:07.924
  STEP: deleting custom profile /go/src/github.com/openshift/cluster-node-tuning-operator/examples/ingress.yaml @ 08/10/24 14:01:08.959
  run command 'oc [delete -n openshift-cluster-node-tuning-operator -f /go/src/github.com/openshift/cluster-node-tuning-operator/examples/ingress.yaml]':
    out=tuned.tuned.openshift.io "openshift-ingress" deleted
    err=
    ret=<nil>
  STEP: waiting for TuneD profile openshift-node on node ip-10-0-14-106.us-west-2.compute.internal @ 08/10/24 14:01:09.312
  STEP: ensuring the default net.ipv4.tcp_tw_reuse value (2) is set in Pod tuned-rn9gc @ 08/10/24 14:01:10.381
  STEP: cluster changes rollback @ 08/10/24 14:01:11.426
  run command 'oc [label pod --overwrite -n openshift-cluster-node-tuning-operator tuned-rn9gc tuned.openshift.io/ingress-]':
    out=label "tuned.openshift.io/ingress" not found.
  pod/tuned-rn9gc not labeled
    err=
    ret=<nil>
  run command 'oc [delete -n openshift-cluster-node-tuning-operator -f /go/src/github.com/openshift/cluster-node-tuning-operator/examples/ingress.yaml]':
    out=
    err=Error from server (NotFound): error when deleting "/go/src/github.com/openshift/cluster-node-tuning-operator/examples/ingress.yaml": tuneds.tuned.openshift.io "openshift-ingress" not found
    ret=exit status 1
• [8.466 seconds]
------------------------------
[basic][rollback] Node Tuning Operator settings rollback TuneD settings rollback without pod restart kernel.shmmni set
/go/src/github.com/openshift/cluster-node-tuning-operator/test/e2e/basic/rollback.go:149
  STEP: getting a list of worker nodes @ 08/10/24 14:01:12.128
  STEP: waiting for TuneD profile openshift-node on node ip-10-0-14-106.us-west-2.compute.internal @ 08/10/24 14:01:12.318
  STEP: checking the pristine state on node ip-10-0-14-106.us-west-2.compute.internal @ 08/10/24 14:01:12.383
  STEP: verification command: [/usr/sbin/sysctl -n kernel.shmmni] @ 08/10/24 14:01:12.383
  STEP: verification expected output: "8192" @ 08/10/24 14:01:12.383
  STEP: verification current output: "4096" @ 08/10/24 14:01:13.481
  STEP: creating custom profile /go/src/github.com/openshift/cluster-node-tuning-operator/test/e2e/testing_manifests/deferred/tuned-basic-00.yaml @ 08/10/24 14:01:13.481
  run command 'oc [create -n openshift-cluster-node-tuning-operator -f /go/src/github.com/openshift/cluster-node-tuning-operator/test/e2e/testing_manifests/deferred/tuned-basic-00.yaml]':
    out=tuned.tuned.openshift.io/ocp-prof-deferred-basic-00 created
    err=
    ret=<nil>
  STEP: waiting for TuneD profile test-shmmni on node ip-10-0-14-106.us-west-2.compute.internal @ 08/10/24 14:01:13.77
  STEP: ensuring the custom worker node profile was set @ 08/10/24 14:01:18.839
  STEP: deleting custom profile /go/src/github.com/openshift/cluster-node-tuning-operator/test/e2e/testing_manifests/deferred/tuned-basic-00.yaml @ 08/10/24 14:01:19.86
  run command 'oc [delete -n openshift-cluster-node-tuning-operator -f /go/src/github.com/openshift/cluster-node-tuning-operator/test/e2e/testing_manifests/deferred/tuned-basic-00.yaml]':
    out=tuned.tuned.openshift.io "ocp-prof-deferred-basic-00" deleted
    err=
    ret=<nil>
  STEP: waiting for TuneD profile openshift-node on node ip-10-0-14-106.us-west-2.compute.internal @ 08/10/24 14:01:20.221
  STEP: ensuring the pristine state is restored on node ip-10-0-14-106.us-west-2.compute.internal @ 08/10/24 14:01:25.29
  STEP: cluster changes rollback @ 08/10/24 14:01:26.314
  run command 'oc [delete -n openshift-cluster-node-tuning-operator -f /go/src/github.com/openshift/cluster-node-tuning-operator/test/e2e/testing_manifests/deferred/tuned-basic-00.yaml]':
    out=
    err=Error from server (NotFound): error when deleting "/go/src/github.com/openshift/cluster-node-tuning-operator/test/e2e/testing_manifests/deferred/tuned-basic-00.yaml": tuneds.tuned.openshift.io "ocp-prof-deferred-basic-00" not found
    ret=exit status 1
• [14.458 seconds]
------------------------------
[basic][default_irq_smp_affinity] Node Tuning Operator set irq default smp affinity irq default smp affinity default_irq_smp_affinity: /proc/irq/default_smp_affinity set
/go/src/github.com/openshift/cluster-node-tuning-operator/test/e2e/basic/default_irq_smp_affinity.go:45
  STEP: getting a list of worker nodes @ 08/10/24 14:01:26.586
  STEP: getting a TuneD Pod running on node ip-10-0-14-106.us-west-2.compute.internal @ 08/10/24 14:01:26.776
  STEP: getting the number of CPUs on ip-10-0-14-106.us-west-2.compute.internal @ 08/10/24 14:01:26.841
  STEP: waiting for TuneD profile openshift-node on node ip-10-0-14-106.us-west-2.compute.internal @ 08/10/24 14:01:27.833
  STEP: getting the original value of /proc/irq/default_smp_affinity @ 08/10/24 14:01:27.897
  tuned-rn9gc has the last nibble of /proc/irq/default_smp_affinity: f
  STEP: labelling node ip-10-0-14-106.us-west-2.compute.internal with label tuned.openshift.io/default-irq-smp-affinity @ 08/10/24 14:01:28.94
  run command 'oc [label node --overwrite ip-10-0-14-106.us-west-2.compute.internal tuned.openshift.io/default-irq-smp-affinity=]':
    out=node/ip-10-0-14-106.us-west-2.compute.internal labeled
    err=
    ret=<nil>
  STEP: creating the custom affinity profile ../testing_manifests/default_irq_smp_affinity0.yaml @ 08/10/24 14:01:29.364
  run command 'oc [create -n openshift-cluster-node-tuning-operator -f ../testing_manifests/default_irq_smp_affinity0.yaml]':
    out=tuned.tuned.openshift.io/openshift-default-irq-smp-affinity created
    err=
    ret=<nil>
  STEP: ensuring the correct value of d was set in the last nibble of /proc/irq/default_smp_affinity @ 08/10/24 14:01:29.645
  STEP: applying the custom affinity profile ../testing_manifests/default_irq_smp_affinity1.yaml @ 08/10/24 14:01:30.672
  run command 'oc [apply -n openshift-cluster-node-tuning-operator -f ../testing_manifests/default_irq_smp_affinity1.yaml]':
    out=tuned.tuned.openshift.io/openshift-default-irq-smp-affinity configured
    err=Warning: resource tuneds/openshift-default-irq-smp-affinity is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by oc apply. oc apply should only be used on resources created declaratively by either oc create --save-config or oc apply. The missing annotation will be patched automatically.
    ret=<nil>
  STEP: ensuring the correct value of 2 was set in the last nibble of /proc/irq/default_smp_affinity @ 08/10/24 14:01:33.535
  STEP: deleting the custom affinity profile ../testing_manifests/default_irq_smp_affinity0.yaml @ 08/10/24 14:01:34.551
  run command 'oc [delete -n openshift-cluster-node-tuning-operator -f ../testing_manifests/default_irq_smp_affinity0.yaml]':
    out=tuned.tuned.openshift.io "openshift-default-irq-smp-affinity" deleted
    err=
    ret=<nil>
  STEP: ensuring the original value of f was set in the last nibble of /proc/irq/default_smp_affinity @ 08/10/24 14:01:34.895
  STEP: removing label tuned.openshift.io/default-irq-smp-affinity from node ip-10-0-14-106.us-west-2.compute.internal @ 08/10/24 14:01:35.939
  run command 'oc [label node --overwrite ip-10-0-14-106.us-west-2.compute.internal tuned.openshift.io/default-irq-smp-affinity-]':
    out=node/ip-10-0-14-106.us-west-2.compute.internal labeled
    err=
    ret=<nil>
  STEP: cluster changes rollback @ 08/10/24 14:01:36.372
  run command 'oc [label node --overwrite ip-10-0-14-106.us-west-2.compute.internal tuned.openshift.io/default-irq-smp-affinity-]':
    out=label "tuned.openshift.io/default-irq-smp-affinity" not found.
  node/ip-10-0-14-106.us-west-2.compute.internal not labeled
    err=
    ret=<nil>
  run command 'oc [delete -n openshift-cluster-node-tuning-operator -f ../testing_manifests/default_irq_smp_affinity0.yaml]':
    out=
    err=Error from server (NotFound): error when deleting "../testing_manifests/default_irq_smp_affinity0.yaml": tuneds.tuned.openshift.io "openshift-default-irq-smp-affinity" not found
    ret=exit status 1
• [10.508 seconds]
------------------------------
[basic][default_node_sysctl] Node Tuning Operator default profile set net.ipv4.neigh.default.gc_thresh1 set
/go/src/github.com/openshift/cluster-node-tuning-operator/test/e2e/basic/default_node_sysctl.go:21
  STEP: getting a list of worker nodes @ 08/10/24 14:01:37.094
  STEP: getting a TuneD Pod running on node ip-10-0-14-106.us-west-2.compute.internal @ 08/10/24 14:01:37.283
  STEP: waiting for TuneD profile openshift-node on node ip-10-0-14-106.us-west-2.compute.internal @ 08/10/24 14:01:37.348
  STEP: ensuring the default worker node profile was set @ 08/10/24 14:01:37.413
• [1.287 seconds]
------------------------------
[basic][available] Node Tuning Operator availability ClusterOperator/node-tuning available and not degraded
/go/src/github.com/openshift/cluster-node-tuning-operator/test/e2e/basic/available.go:29
  STEP: waiting for ClusterOperator/node-tuning available @ 08/10/24 14:01:38.38
  STEP: waiting for ClusterOperator/node-tuning not degraded @ 08/10/24 14:01:38.445
• [0.129 seconds]
------------------------------
[basic][available] Node Tuning Operator availability Tuned/default exists
/go/src/github.com/openshift/cluster-node-tuning-operator/test/e2e/basic/available.go:39
  STEP: waiting for Tuned/default existence @ 08/10/24 14:01:38.509
• [0.064 seconds]
------------------------------
[basic][tuned_builtin_expand] Node Tuning Operator custom profile, TuneD built-in expansion custom profile: built-in expansion vm.nr_hugepages set
/go/src/github.com/openshift/cluster-node-tuning-operator/test/e2e/basic/tuned_builtin_expand.go:41
  STEP: getting a list of worker nodes @ 08/10/24 14:01:38.573
  STEP: getting a TuneD Pod running on node ip-10-0-14-106.us-west-2.compute.internal @ 08/10/24 14:01:38.763
  STEP: waiting for TuneD profile openshift-node on node ip-10-0-14-106.us-west-2.compute.internal @ 08/10/24 14:01:38.828
  STEP: getting the current value of vm.nr_hugepages in Pod tuned-rn9gc @ 08/10/24 14:01:38.892
  STEP: labelling node ip-10-0-14-106.us-west-2.compute.internal with label tuned.openshift.io/tuned-built-in @ 08/10/24 14:01:39.911
  run command 'oc [label node --overwrite ip-10-0-14-106.us-west-2.compute.internal tuned.openshift.io/tuned-built-in=]':
    out=node/ip-10-0-14-106.us-west-2.compute.internal labeled
    err=
    ret=<nil>
  STEP: creating the custom profile ../../../examples/hugepages.yaml @ 08/10/24 14:01:40.34
  run command 'oc [create -n openshift-cluster-node-tuning-operator -f ../../../examples/hugepages.yaml]':
    out=tuned.tuned.openshift.io/openshift-hugepages created
    err=
    ret=<nil>
  STEP: creating the custom profile ../testing_manifests/tuned_builtin_expand.yaml @ 08/10/24 14:01:40.634
  run command 'oc [create -n openshift-cluster-node-tuning-operator -f ../testing_manifests/tuned_builtin_expand.yaml]':
    out=tuned.tuned.openshift.io/openshift-tuned-builtin created
    err=
    ret=<nil>
  STEP: ensuring the custom worker node profile was set @ 08/10/24 14:01:40.908
  STEP: deleting the custom profile ../testing_manifests/tuned_builtin_expand.yaml @ 08/10/24 14:01:41.946
  run command 'oc [delete -n openshift-cluster-node-tuning-operator -f ../testing_manifests/tuned_builtin_expand.yaml]':
    out=tuned.tuned.openshift.io "openshift-tuned-builtin" deleted
    err=
    ret=<nil>
  STEP: deleting the custom profile ../../../examples/hugepages.yaml @ 08/10/24 14:01:42.296
  run command 'oc [delete -n openshift-cluster-node-tuning-operator -f ../../../examples/hugepages.yaml]':
    out=tuned.tuned.openshift.io "openshift-hugepages" deleted
    err=
    ret=<nil>
  STEP: ensuring the original vm.nr_hugepages value (0) is set in Pod tuned-rn9gc @ 08/10/24 14:01:42.656
  STEP: removing label tuned.openshift.io/tuned-built-in from node ip-10-0-14-106.us-west-2.compute.internal @ 08/10/24 14:01:43.628
  run command 'oc [label node --overwrite ip-10-0-14-106.us-west-2.compute.internal tuned.openshift.io/tuned-built-in-]':
    out=node/ip-10-0-14-106.us-west-2.compute.internal labeled
    err=
    ret=<nil>
  STEP: cluster changes rollback @ 08/10/24 14:01:44.029
  run command 'oc [label node --overwrite ip-10-0-14-106.us-west-2.compute.internal tuned.openshift.io/tuned-built-in-]':
    out=label "tuned.openshift.io/tuned-built-in" not found.
  node/ip-10-0-14-106.us-west-2.compute.internal not labeled
    err=
    ret=<nil>
  run command 'oc [delete -n openshift-cluster-node-tuning-operator -f ../testing_manifests/tuned_builtin_expand.yaml]':
    out=
    err=Error from server (NotFound): error when deleting "../testing_manifests/tuned_builtin_expand.yaml": tuneds.tuned.openshift.io "openshift-tuned-builtin" not found
    ret=exit status 1
  run command 'oc [delete -n openshift-cluster-node-tuning-operator -f ../../../examples/hugepages.yaml]':
    out=
    err=Error from server (NotFound): error when deleting "../../../examples/hugepages.yaml": tuneds.tuned.openshift.io "openshift-hugepages" not found
    ret=exit status 1
• [6.440 seconds]
------------------------------
[basic][metrics] Node Tuning Operator certificate rotation TLS certificate rotation delete certificate Secret and check that the server restarts with latest certificate
/go/src/github.com/openshift/cluster-node-tuning-operator/test/e2e/basic/metrics_cert_rotation.go:20
  STEP: deleting node-tuning-operator-tls Secret to trigger certificate rotation @ 08/10/24 14:01:45.014
  run command 'oc [delete -n openshift-cluster-node-tuning-operator secret/node-tuning-operator-tls]':
    out=secret "node-tuning-operator-tls" deleted
    err=
    ret=<nil>
  STEP: deleting node-tuning-operator-tls Secret a second time @ 08/10/24 14:01:45.483
  run command 'oc [delete -n openshift-cluster-node-tuning-operator secret/node-tuning-operator-tls]':
    out=secret "node-tuning-operator-tls" deleted
    err=
    ret=<nil>
  Error getting secret/node-tuning-operator-tls. May not exist yet. Err: secrets "node-tuning-operator-tls" not found
  STEP: getting a list of worker nodes @ 08/10/24 14:01:47.956
  STEP: getting a TuneD Pod running on node ip-10-0-14-106.us-west-2.compute.internal @ 08/10/24 14:01:48.094
  STEP: getting cluster-node-tuning-operator Pod @ 08/10/24 14:01:48.16
  STEP: checking if server TLS certificate matches TLS certificate in Secret @ 08/10/24 14:01:48.225
• [34.320 seconds]
------------------------------
[basic][tuned_errors_and_recovery] Cause TuneD daemon errors and recover TuneD daemon errors and recovery Cause TuneD daemon errors on invalid profile load and recover after the profile deletion
/go/src/github.com/openshift/cluster-node-tuning-operator/test/e2e/basic/tuned_errors_and_recovery.go:47
  STEP: getting a list of worker nodes @ 08/10/24 14:02:19.334
  STEP: getting a TuneD Pod running on node ip-10-0-14-106.us-west-2.compute.internal @ 08/10/24 14:02:19.523
  STEP: labelling node ip-10-0-14-106.us-west-2.compute.internal with label tuned.openshift.io/cause-tuned-failure @ 08/10/24 14:02:19.589
  run command 'oc [label node --overwrite ip-10-0-14-106.us-west-2.compute.internal tuned.openshift.io/cause-tuned-failure=]':
    out=node/ip-10-0-14-106.us-west-2.compute.internal labeled
    err=
    ret=<nil>
  STEP: waiting for ClusterOperator/node-tuning condition Available reason AsExpected @ 08/10/24 14:02:20.015
  STEP: creating the custom profile ../testing_manifests/cause_tuned_failure.yaml @ 08/10/24 14:02:20.081
  run command 'oc [create -n openshift-cluster-node-tuning-operator -f ../testing_manifests/cause_tuned_failure.yaml]':
    out=tuned.tuned.openshift.io/openshift-cause-tuned-failure created
    err=
    ret=<nil>
  STEP: waiting for ClusterOperator/node-tuning condition Available reason ProfileDegraded @ 08/10/24 14:02:20.363
  STEP: creating the custom profile ../testing_manifests/dummy.yaml @ 08/10/24 14:02:25.429
  run command 'oc [create -n openshift-cluster-node-tuning-operator -f ../testing_manifests/dummy.yaml]':
    out=tuned.tuned.openshift.io/openshift-dummy created
    err=
    ret=<nil>
  STEP: waiting for ClusterOperator/node-tuning condition Available reason AsExpected @ 08/10/24 14:02:25.715
  STEP: deleting the custom profile ../testing_manifests/cause_tuned_failure.yaml @ 08/10/24 14:02:25.783
  run command 'oc [delete -n openshift-cluster-node-tuning-operator -f ../testing_manifests/cause_tuned_failure.yaml]':
    out=tuned.tuned.openshift.io "openshift-cause-tuned-failure" deleted
    err=
    ret=<nil>
  STEP: deleting the custom profile ../testing_manifests/dummy.yaml @ 08/10/24 14:02:26.137
  run command 'oc [delete -n openshift-cluster-node-tuning-operator -f ../testing_manifests/dummy.yaml]':
    out=tuned.tuned.openshift.io "openshift-dummy" deleted
    err=
    ret=<nil>
  STEP: waiting for ClusterOperator/node-tuning condition Available reason AsExpected @ 08/10/24 14:02:26.478
  STEP: removing label tuned.openshift.io/cause-tuned-failure from node ip-10-0-14-106.us-west-2.compute.internal @ 08/10/24 14:02:26.542
  run command 'oc [label node --overwrite ip-10-0-14-106.us-west-2.compute.internal tuned.openshift.io/cause-tuned-failure-]':
    out=node/ip-10-0-14-106.us-west-2.compute.internal labeled
    err=
    ret=<nil>
  STEP: cluster changes rollback @ 08/10/24 14:02:26.942
  run command 'oc [label node --overwrite ip-10-0-14-106.us-west-2.compute.internal tuned.openshift.io/cause-tuned-failure-]':
    out=label "tuned.openshift.io/cause-tuned-failure" not found.
  node/ip-10-0-14-106.us-west-2.compute.internal not labeled
    err=
    ret=<nil>
  run command 'oc [delete -n openshift-cluster-node-tuning-operator -f ../testing_manifests/cause_tuned_failure.yaml]':
    out=
    err=Error from server (NotFound): error when deleting "../testing_manifests/cause_tuned_failure.yaml": tuneds.tuned.openshift.io "openshift-cause-tuned-failure" not found
    ret=exit status 1
  run command 'oc [delete -n openshift-cluster-node-tuning-operator -f ../testing_manifests/dummy.yaml]':
    out=
    err=Error from server (NotFound): error when deleting "../testing_manifests/dummy.yaml": tuneds.tuned.openshift.io "openshift-dummy" not found
    ret=exit status 1
  run command 'oc [exec -n openshift-cluster-node-tuning-operator tuned-rn9gc -- rm -rf /etc/tuned/openshift-dummy]':
    out=
    err=
    ret=<nil>
• [9.539 seconds]
------------------------------
[basic][sysctl_d_override] Node Tuning Operator /etc/sysctl.d/*.conf override sysctl.d override net.ipv4.neigh.default.gc_thresh1 set
/go/src/github.com/openshift/cluster-node-tuning-operator/test/e2e/basic/sysctl_d_override.go:48
  STEP: getting a list of worker nodes @ 08/10/24 14:02:28.872
  STEP: getting a TuneD Pod running on node ip-10-0-14-106.us-west-2.compute.internal @ 08/10/24 14:02:29.062
  STEP: waiting for TuneD profile openshift-node on node ip-10-0-14-106.us-west-2.compute.internal @ 08/10/24 14:02:29.127
  STEP: getting the current value of net.ipv4.neigh.default.gc_thresh1 in Pod tuned-rn9gc @ 08/10/24 14:02:29.192
  STEP: writing /host/etc/sysctl.d/zzz.conf override file on the host with net.ipv4.neigh.default.gc_thresh1=256 @ 08/10/24 14:02:30.217
  run command 'oc [exec -n openshift-cluster-node-tuning-operator tuned-rn9gc -- sh -c echo net.ipv4.neigh.default.gc_thresh1=256 > /host/etc/sysctl.d/zzz.conf; sync /host/etc/sysctl.d/zzz.conf]':
    out=
    err=
    ret=<nil>
  run command 'oc [rsh -n openshift-cluster-node-tuning-operator tuned-rn9gc cat /host/etc/sysctl.d/zzz.conf]':
    out=net.ipv4.neigh.default.gc_thresh1=256
    err=
    ret=<nil>
  STEP: deleting Pod tuned-rn9gc @ 08/10/24 14:02:32.218
  run command 'oc [delete -n openshift-cluster-node-tuning-operator pod tuned-rn9gc --wait]':
    out=pod "tuned-rn9gc" deleted
    err=
    ret=<nil>
  STEP: waiting for a new TuneD Pod to be ready on node ip-10-0-14-106.us-west-2.compute.internal @ 08/10/24 14:02:32.996
  STEP: ensuring new net.ipv4.neigh.default.gc_thresh1 value (256) is set in Pod tuned-n77w2 @ 08/10/24 14:02:33.061
  [FAILED] in [It] - /go/src/github.com/openshift/cluster-node-tuning-operator/test/e2e/basic/sysctl_d_override.go:98 @ 08/10/24 14:07:34.103
  STEP: cluster changes rollback @ 08/10/24 14:07:34.103
  run command 'oc [label node --overwrite ip-10-0-14-106.us-west-2.compute.internal tuned.openshift.io/sysctl-override-]':
    out=label "tuned.openshift.io/sysctl-override" not found.
  node/ip-10-0-14-106.us-west-2.compute.internal not labeled
    err=
    ret=<nil>
  run command 'oc [exec -n openshift-cluster-node-tuning-operator tuned-n77w2 -- rm /host/etc/sysctl.d/zzz.conf]':
    out=
    err=
    ret=<nil>
  run command 'oc [delete -n openshift-cluster-node-tuning-operator -f ../testing_manifests/sysctl_override.yaml]':
    out=
    err=Error from server (NotFound): error when deleting "../testing_manifests/sysctl_override.yaml": tuneds.tuned.openshift.io "openshift-profile" not found
    ret=exit status 1
• [FAILED] [306.897 seconds]
[basic][sysctl_d_override] Node Tuning Operator /etc/sysctl.d/*.conf override sysctl.d override [It] net.ipv4.neigh.default.gc_thresh1 set
/go/src/github.com/openshift/cluster-node-tuning-operator/test/e2e/basic/sysctl_d_override.go:48

  [FAILED] Unexpected error:
      <*errors.errorString | 0xc00022e8e0>: 
      sysctl net.ipv4.neigh.default.gc_thresh1=8192 in Pod tuned-n77w2, expected 256: command [sysctl -n net.ipv4.neigh.default.gc_thresh1] outputs 8192 (leading/trailing whitespace trimmed) in Pod tuned-n77w2, expected 256 (waited 5m1.04114747s): <nil>
      {
          s: "sysctl net.ipv4.neigh.default.gc_thresh1=8192 in Pod tuned-n77w2, expected 256: command [sysctl -n net.ipv4.neigh.default.gc_thresh1] outputs 8192 (leading/trailing whitespace trimmed) in Pod tuned-n77w2, expected 256 (waited 5m1.04114747s): <nil>",
      }
  occurred
  In [It] at: /go/src/github.com/openshift/cluster-node-tuning-operator/test/e2e/basic/sysctl_d_override.go:98 @ 08/10/24 14:07:34.103
------------------------------
SSSSS

Summarizing 1 Failure:
  [FAIL] [basic][sysctl_d_override] Node Tuning Operator /etc/sysctl.d/*.conf override sysctl.d override [It] net.ipv4.neigh.default.gc_thresh1 set
  /go/src/github.com/openshift/cluster-node-tuning-operator/test/e2e/basic/sysctl_d_override.go:98

Ran 10 of 15 Specs in 392.108 seconds
FAIL! -- 9 Passed | 1 Failed | 0 Pending | 5 Skipped
--- FAIL: TestNodeTuningOperator (392.11s)
FAIL
FAIL	github.com/openshift/cluster-node-tuning-operator/test/e2e/basic	392.124s
FAIL
make: *** [Makefile:103: test-e2e] Error 1
{"component":"entrypoint","error":"wrapped process failed: exit status 2","file":"sigs.k8s.io/prow/pkg/entrypoint/run.go:84","func":"sigs.k8s.io/prow/pkg/entrypoint.Options.internalRun","level":"error","msg":"Error executing test process","severity":"error","time":"2024-08-10T14:07:35Z"}
error: failed to execute wrapped command: exit status 2 
INFO[2024-08-10T14:07:36Z] Step e2e-aws-operator-test failed after 7m22s. 
INFO[2024-08-10T14:07:36Z] Step phase test failed after 7m22s.          
INFO[2024-08-10T14:07:36Z] Running multi-stage phase post               
INFO[2024-08-10T14:07:36Z] Running step e2e-aws-operator-gather-aws-console. 
INFO[2024-08-10T14:07:57Z] Step e2e-aws-operator-gather-aws-console succeeded after 21s. 
INFO[2024-08-10T14:07:57Z] Running step e2e-aws-operator-gather-must-gather. 
INFO[2024-08-10T14:09:39Z] Step e2e-aws-operator-gather-must-gather succeeded after 1m41s. 
INFO[2024-08-10T14:09:39Z] Running step e2e-aws-operator-gather-extra.  
INFO[2024-08-10T14:11:20Z] Step e2e-aws-operator-gather-extra succeeded after 1m40s. 
INFO[2024-08-10T14:11:20Z] Running step e2e-aws-operator-gather-audit-logs. 
INFO[2024-08-10T14:11:44Z] Step e2e-aws-operator-gather-audit-logs succeeded after 24s. 
INFO[2024-08-10T14:11:44Z] Running step e2e-aws-operator-ipi-deprovision-deprovision. 
INFO[2024-08-10T14:16:56Z] Step e2e-aws-operator-ipi-deprovision-deprovision succeeded after 5m11s. 
INFO[2024-08-10T14:16:56Z] Step phase post succeeded after 9m20s.       
INFO[2024-08-10T14:16:56Z] Releasing ip pool leases for test e2e-aws-operator 
INFO[2024-08-10T14:16:56Z] Releasing leases for test e2e-aws-operator   
INFO[2024-08-10T14:16:56Z] Ran for 1h5m49s                              
ERRO[2024-08-10T14:16:56Z] Some steps failed:                           
ERRO[2024-08-10T14:16:56Z] 
  * could not run steps: step e2e-aws-operator failed: "e2e-aws-operator" test steps failed: "e2e-aws-operator" pod "e2e-aws-operator-test" failed: could not watch pod: the pod ci-op-q69b2z86/e2e-aws-operator-test failed after 7m23s (failed containers: test): ContainerFailed one or more containers exited

Container test exited with code 2, reason Error
---
ests/sysctl_override.yaml": tuneds.tuned.openshift.io "openshift-profile" not found
    ret=exit status 1
• [FAILED] [306.897 seconds]
[basic][sysctl_d_override] Node Tuning Operator /etc/sysctl.d/*.conf override sysctl.d override [It] net.ipv4.neigh.default.gc_thresh1 set
/go/src/github.com/openshift/cluster-node-tuning-operator/test/e2e/basic/sysctl_d_override.go:48

  [FAILED] Unexpected error:
      <*errors.errorString | 0xc00022e8e0>: 
      sysctl net.ipv4.neigh.default.gc_thresh1=8192 in Pod tuned-n77w2, expected 256: command [sysctl -n net.ipv4.neigh.default.gc_thresh1] outputs 8192 (leading/trailing whitespace trimmed) in Pod tuned-n77w2, expected 256 (waited 5m1.04114747s): <nil>
      {
          s: "sysctl net.ipv4.neigh.default.gc_thresh1=8192 in Pod tuned-n77w2, expected 256: command [sysctl -n net.ipv4.neigh.default.gc_thresh1] outputs 8192 (leading/trailing whitespace trimmed) in Pod tuned-n77w2, expected 256 (waited 5m1.04114747s): <nil>",
      }
  occurred
  In [It] at: /go/src/github.com/openshift/cluster-node-tuning-operator/test/e2e/basic/sysctl_d_override.go:98 @ 08/10/24 14:07:34.103
------------------------------
SSSSS

Summarizing 1 Failure:
  [FAIL] [basic][sysctl_d_override] Node Tuning Operator /etc/sysctl.d/*.conf override sysctl.d override [It] net.ipv4.neigh.default.gc_thresh1 set
  /go/src/github.com/openshift/cluster-node-tuning-operator/test/e2e/basic/sysctl_d_override.go:98

Ran 10 of 15 Specs in 392.108 seconds
FAIL! -- 9 Passed | 1 Failed | 0 Pending | 5 Skipped
--- FAIL: TestNodeTuningOperator (392.11s)
FAIL
FAIL	github.com/openshift/cluster-node-tuning-operator/test/e2e/basic	392.124s
FAIL
make: *** [Makefile:103: test-e2e] Error 1
{"component":"entrypoint","error":"wrapped process failed: exit status 2","file":"sigs.k8s.io/prow/pkg/entrypoint/run.go:84","func":"sigs.k8s.io/prow/pkg/entrypoint.Options.internalRun","level":"error","msg":"Error executing test process","severity":"error","time":"2024-08-10T14:07:35Z"}
error: failed to execute wrapped command: exit status 2
---
Link to step on registry info site: https://steps.ci.openshift.org/reference/test
Link to job on registry info site: https://steps.ci.openshift.org/job?org=openshift&repo=cluster-node-tuning-operator&branch=master&test=e2e-aws-operator 
INFO[2024-08-10T14:16:56Z] Reporting job state 'failed' with reason 'executing_graph:step_failed:utilizing_lease:executing_test:utilizing_ip_pool:executing_test:executing_multi_stage_test'