Jump to content

Server Admin Log

From Wikitech
(Redirected from Server admin log)

2026-10-31

  • 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
  • 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
  • 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
  • 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
  • 02:08 mwpresync@deploy2003: Finished scap build-images: Publishing wmf/next image (duration: 07m 22s)
  • 02:01 mwpresync@deploy2003: Started scap build-images: Publishing wmf/next image
  • 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
  • 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync

2026-10-30

  • 21:22 krinkle@deploy2003: Finished scap sync-world: Backport for Batch file retrievals for search result thumbnails (T441253) (1.48.0-wmf.1) (duration: 25m 22s)
  • 21:14 krinkle@deploy2003: krinkle: Continuing with deployment
  • 21:01 krinkle@deploy2003: krinkle: Backport for Batch file retrievals for search result thumbnails (T441253) (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 20:57 krinkle@deploy2003: Started scap sync-world: Backport for Batch file retrievals for search result thumbnails (T441253) (1.48.0-wmf.1)
  • 19:44 inflatador: bking@build2004 docker-pkg -c /etc/production-images/config.yaml build images/ --select '*flink* T443032
  • 18:50 cmooney@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1250.eqiad.wmnet
  • 18:50 cmooney@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1250.eqiad.wmnet
  • 18:28 cmooney@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-a6-codfw
  • 18:28 cmooney@cumin1004: START - Cookbook sre.network.tls for network device lsw1-a6-codfw
  • 18:22 cmooney@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-a5-codfw
  • 18:22 cmooney@cumin1004: START - Cookbook sre.network.tls for network device lsw1-a5-codfw
  • 18:22 cmooney@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-b5-codfw
  • 18:21 cmooney@cumin1004: START - Cookbook sre.network.tls for network device lsw1-b5-codfw
  • 16:09 cmooney@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device ssw1-f1-eqiad
  • 16:09 cmooney@cumin1004: START - Cookbook sre.network.tls for network device ssw1-f1-eqiad
  • 16:09 cmooney@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-a4-codfw
  • 16:08 cmooney@cumin1004: START - Cookbook sre.network.tls for network device lsw1-a4-codfw
  • 16:08 cmooney@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-b2-codfw
  • 16:08 cmooney@cumin1004: START - Cookbook sre.network.tls for network device lsw1-b2-codfw
  • 16:08 cmooney@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-b6-codfw
  • 16:08 cmooney@cumin1004: START - Cookbook sre.network.tls for network device lsw1-b6-codfw
  • 16:08 cmooney@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-b8-codfw
  • 16:08 cmooney@cumin1004: START - Cookbook sre.network.tls for network device lsw1-b8-codfw
  • 16:03 otto@deploy2003: helmfile [codfw] DONE helmfile.d/services/eventgate-main: sync
  • 16:02 otto@deploy2003: helmfile [codfw] START helmfile.d/services/eventgate-main: sync
  • 16:02 otto@deploy2003: helmfile [eqiad] DONE helmfile.d/services/eventgate-main: sync
  • 16:02 otto@deploy2003: helmfile [eqiad] START helmfile.d/services/eventgate-main: sync
  • 16:02 ottomata: roll restarting eventgate main to pick up EventStreamConfig: Add revision_html_feature_counts_change.dev0 stream (operations/mediawiki-config~1360516)
  • 16:01 otto@deploy2003: helmfile [staging] DONE helmfile.d/services/eventgate-main: sync
  • 16:01 otto@deploy2003: helmfile [staging] START helmfile.d/services/eventgate-main: sync
  • 15:16 javiermonton@deploy2003: Finished scap deploy-service pageview-trending-relative-next: Test spiderpig with Flink checks. This is a staging release. (duration: 02m 17s)
  • 15:14 javiermonton@deploy2003: scap deploy-service pageview-trending-relative-next: [PRODUCTION/DEFAULT] dse-k8s-eqiad Deploying pageview-trending-relative-next
  • 15:14 javiermonton@deploy2003: Started scap deploy-service pageview-trending-relative-next: Test spiderpig with Flink checks. This is a staging release.
  • 15:04 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply
  • 15:04 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply
  • 15:03 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply
  • 15:02 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply
  • 14:47 kevinbazira@deploy2003: helmfile [ml-serve-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' .
  • 14:46 kevinbazira@deploy2003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'tts-section-generator' for release 'main' .
  • 14:44 kevinbazira@deploy2003: helmfile [ml-staging-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' .
  • 14:43 cmooney@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1240.eqiad.wmnet
  • 14:43 cmooney@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1240.eqiad.wmnet
  • 14:35 otto@deploy2003: Finished scap sync-world: Backport for EventStreamConfig: Add revision_html_feature_counts_change.dev0 stream (T436695) (duration: 36m 47s)
  • 14:23 otto@deploy2003: otto: Continuing with deployment
  • 14:21 otto@deploy2003: otto: Backport for EventStreamConfig: Add revision_html_feature_counts_change.dev0 stream (T436695) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 14:15 cmooney@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1240.eqiad.wmnet
  • 14:14 topranks: depool wikikube-worker1240 to trial 1G -> 10G migration process for eqiad A/B wikikubes T440696
  • 14:14 cmooney@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1240.eqiad.wmnet
  • 13:58 otto@deploy2003: Started scap sync-world: Backport for EventStreamConfig: Add revision_html_feature_counts_change.dev0 stream (T436695)
  • 13:40 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user hokwelum
  • 13:39 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user hokwelum
  • 13:39 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user akosiaris
  • 13:39 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user akosiaris
  • 13:38 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user sguebo
  • 13:38 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user sguebo
  • 12:50 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user iflorez
  • 12:48 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user iflorez
  • 12:44 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user niharika29
  • 12:44 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user niharika29
  • 12:43 gkyziridis@deploy2003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revise-tone-task-generator' for release 'main' .
  • 12:43 gkyziridis@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revise-tone-task-generator' for release 'main' .
  • 12:34 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetboard1003.eqiad.wmnet
  • 12:30 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host puppetboard1003.eqiad.wmnet
  • 12:29 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetboard2003.codfw.wmnet
  • 12:26 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host puppetboard2003.codfw.wmnet
  • 12:21 fnegri@deploy2003: helmfile [codfw] DONE helmfile.d/services/toolhub: apply
  • 12:20 fnegri@deploy2003: helmfile [codfw] START helmfile.d/services/toolhub: apply
  • 12:20 ozge@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm-batch' for release 'llmisvc' .
  • 12:20 ozge@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm-batch' for release 'main' .
  • 12:20 fnegri@deploy2003: helmfile [eqiad] DONE helmfile.d/services/toolhub: apply
  • 12:20 ozge@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm-batch' for release 'llmisvc' .
  • 12:19 fnegri@deploy2003: helmfile [eqiad] START helmfile.d/services/toolhub: apply
  • 12:18 fnegri@deploy2003: helmfile [staging] DONE helmfile.d/services/toolhub: apply
  • 12:18 XioNoX: asw1-bw27-esams# delete protocols bgp group PyBal multipath - T401832
  • 12:17 fnegri@deploy2003: helmfile [staging] START helmfile.d/services/toolhub: apply
  • 12:13 ozge@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm-batch' for release 'main' .
  • 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
  • 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
  • 11:56 jelto@cumin1004: END (FAIL) - Cookbook sre.gitlab.upgrade (exit_code=99) on GitLab host gitlab1004.wikimedia.org with reason: version upgrade
  • 11:34 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1298.eqiad.wmnet
  • 11:33 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1298.eqiad.wmnet
  • 11:33 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1300.eqiad.wmnet
  • 11:33 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1300.eqiad.wmnet
  • 11:32 jelto@cumin1004: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: version upgrade
  • 11:31 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker1300.eqiad.wmnet
  • 11:29 jelto@cumin1004: END (FAIL) - Cookbook sre.gitlab.upgrade (exit_code=99) on GitLab host gitlab1004.wikimedia.org with reason: version upgrade
  • 11:29 jelto@cumin1004: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: version upgrade
  • 11:25 jelto@cumin1004: END (FAIL) - Cookbook sre.gitlab.upgrade (exit_code=99) on GitLab host gitlab1004.wikimedia.org with reason: version upgrade
  • 11:24 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker1300.eqiad.wmnet
  • 11:22 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker1298.eqiad.wmnet
  • 11:19 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P{lvs3010.*} and A:liberica
  • 11:18 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin config_reloading P{lvs3010.*} and A:liberica
  • 11:15 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P{lvs3008.*} and A:liberica
  • 11:15 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker1298.eqiad.wmnet
  • 11:14 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin config_reloading P{lvs3008.*} and A:liberica
  • 11:05 jelto@cumin1004: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: version upgrade
  • 11:04 jelto@cumin1004: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab2002.wikimedia.org with reason: version upgrade
  • 11:03 vgutierrez@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host lvs3008.esams.wmnet with OS trixie
  • 10:54 jelto@cumin1004: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab2002.wikimedia.org with reason: version upgrade
  • 10:53 jelto@cumin1004: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1003.wikimedia.org with reason: version upgrade
  • 10:44 vgutierrez@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs3008.esams.wmnet with reason: host reimage
  • 10:42 jelto@cumin1004: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: version upgrade
  • 10:40 vgutierrez@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on lvs3008.esams.wmnet with reason: host reimage
  • 10:37 vgutierrez@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs3008.esams.wmnet (remove config only)
  • 10:37 vgutierrez@cumin1004: START - Cookbook sre.hosts.dhcp for host lvs3008.esams.wmnet (remove config only)
  • 10:28 cwilliams@deploy2003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
  • 10:21 kevinbazira@deploy1003: helmfile [ml-serve-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' .
  • 10:21 vgutierrez@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs3008.esams.wmnet (push config only)
  • 10:21 vgutierrez@cumin1004: START - Cookbook sre.hosts.dhcp for host lvs3008.esams.wmnet (push config only)
  • 10:20 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'tts-section-generator' for release 'main' .
  • 10:20 vgutierrez@cumin1004: START - Cookbook sre.hosts.reimage for host lvs3008.esams.wmnet with OS trixie
  • 10:18 kevinbazira@deploy1003: helmfile [ml-staging-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' .
  • 10:15 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sretest2010.codfw.wmnet
  • 10:14 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P{lvs3008.*} and A:liberica
  • 10:14 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin config_reloading P{lvs3008.*} and A:liberica
  • 10:08 mvernon@cumin2003: START - Cookbook sre.hosts.reboot-single for host sretest2010.codfw.wmnet
  • 10:05 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P{lvs3009.*} and A:liberica
  • 10:04 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin config_reloading P{lvs3009.*} and A:liberica
  • 09:55 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P{lvs3009.*} and A:liberica
  • 09:55 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin config_reloading P{lvs3009.*} and A:liberica
  • 09:48 vgutierrez@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host lvs3009.esams.wmnet with OS trixie
  • 09:35 slyngshede@deploy2003: Finished scap sync-world: Test deployment to validate deployment server switchover - T436630 (duration: 35m 48s)
  • 09:29 vgutierrez@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs3009.esams.wmnet with reason: host reimage
  • 09:25 vgutierrez@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on lvs3009.esams.wmnet with reason: host reimage
  • 09:22 vgutierrez@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs3009.esams.wmnet (remove config only)
  • 09:22 vgutierrez@cumin1004: START - Cookbook sre.hosts.dhcp for host lvs3009.esams.wmnet (remove config only)
  • 09:03 vgutierrez@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs3009.esams.wmnet (push config only)
  • 09:03 vgutierrez@cumin1004: START - Cookbook sre.hosts.dhcp for host lvs3009.esams.wmnet (push config only)
  • 09:02 vgutierrez@cumin1004: START - Cookbook sre.hosts.reimage for host lvs3009.esams.wmnet with OS trixie
  • 09:00 XioNoX: asw1-bw27-esams# set protocols bgp group PyBal multipath - T401832
  • 08:59 slyngshede@deploy2003: Started scap sync-world: Test deployment to validate deployment server switchover - T436630
  • 08:53 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P{lvs3009.*} and A:liberica
  • 08:53 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin config_reloading P{lvs3009.*} and A:liberica
  • 08:43 slyngs: Moved deployment.eqiad.wmnet to deploy2003.codfw.wmnet - T436630
  • 08:37 slyngshede@dns1004: END - running authdns-update
  • 08:34 slyngshede@dns1004: START - running authdns-update
  • 08:32 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P{lvs3010.*} and A:liberica
  • 08:32 slyngshede@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on releases2003.codfw.wmnet,releases1003.eqiad.wmnet with reason: Deployment server switchover
  • 08:32 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin config_reloading P{lvs3010.*} and A:liberica
  • 08:11 kevinbazira@deploy1003: helmfile [ml-serve-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' .
  • 08:10 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'tts-section-generator' for release 'main' .
  • 08:08 kevinbazira@deploy1003: helmfile [ml-staging-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' .
  • 05:06 ryankemper: T442431 [airflow-experiment-platform] completed upgrade to Airflow 3.3; unpaused its DAGs and verified successful growthbook ldap sync dag runs for both staging and prod. this is done
  • 04:50 ryankemper@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-experiment-platform: apply
  • 04:49 ryankemper@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-experiment-platform: apply
  • 04:36 ryankemper: T442431 [airflow-experiment-platform] starting upgrade to Airflow 3.3; pausing its DAGs
  • 02:09 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 51s)
  • 02:01 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image
  • 00:00 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
  • 00:00 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
  • 00:00 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
  • 00:00 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: sync

2026-10-29

  • 23:47 Dreamy_Jazz: Running `foreachwikiindblist group2.dblist cleanupWatchlistTemporaryAccountRows.php` for T439059
  • 23:13 kemayo@deploy1003: Finished scap sync-world: Backport for Remove wgVisualEditorDisableForAnons for enwiki (T442933), Continue to serve config for cached js (T430970) (1.48.0-wmf.1) (duration: 36m 48s)
  • 23:00 kemayo@deploy1003: kemayo, arlolra: Continuing with deployment
  • 22:58 kemayo@deploy1003: kemayo, arlolra: Backport for Remove wgVisualEditorDisableForAnons for enwiki (T442933), Continue to serve config for cached js (T430970) (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 22:36 kemayo@deploy1003: Started scap sync-world: Backport for Remove wgVisualEditorDisableForAnons for enwiki (T442933), Continue to serve config for cached js (T430970) (1.48.0-wmf.1)
  • 22:27 maryum: Deployed security fix for T440122, T439876, T442478, and T442693
  • 21:28 cwilliams@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool pc2017: Maintenance
  • 21:28 cwilliams@cumin1004: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0)
  • 21:28 cwilliams@cumin1004: START - Cookbook sre.mysql.parsercache
  • 21:27 cwilliams@cumin1004: START - Cookbook sre.mysql.pool pool pc2017: Maintenance
  • 20:55 cwilliams@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on pc2017.codfw.wmnet with reason: Alerting
  • 20:51 cwilliams@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool pc2017: Maintenance
  • 20:51 cwilliams@cumin1004: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0)
  • 20:51 cwilliams@cumin1004: START - Cookbook sre.mysql.parsercache
  • 20:51 cwilliams@cumin1004: START - Cookbook sre.mysql.depool depool pc2017: Maintenance
  • 20:37 dancy@deploy1003: Unlocked for deployment [MediaWiki]: Holding until database problems are resolved (duration: 24m 55s)
  • 20:12 dancy@deploy1003: Locking from deployment [MediaWiki]: Holding until database problems are resolved
  • 20:09 dancy@deploy1003: Failed scap sync-world: Backport for listTaskCounts: Do not send negative task counts to stats (T394969) (1.48.0-wmf.1) (duration: 12m 25s)
  • 20:08 cscott@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
  • 20:08 cscott@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
  • 20:07 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/proton: apply
  • 20:06 dancy@deploy1003: Rolling back deployment
  • 20:06 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/proton: apply
  • 20:05 jforrester@deploy1003: helmfile [eqiad] DONE helmfile.d/services/proton: apply
  • 20:04 jforrester@deploy1003: helmfile [eqiad] START helmfile.d/services/proton: apply
  • 20:03 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/proton: apply
  • 20:03 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/proton: apply
  • 20:02 dancy@deploy1003: dancy: Continuing with deployment
  • 20:02 cscott@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
  • 20:01 cscott@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
  • 20:01 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/proton: apply
  • 20:01 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/proton: apply
  • 20:01 dancy@deploy1003: dancy: Backport for listTaskCounts: Do not send negative task counts to stats (T394969) (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 19:57 dancy@deploy1003: Started scap sync-world: Backport for listTaskCounts: Do not send negative task counts to stats (T394969) (1.48.0-wmf.1)
  • 19:56 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/proton: apply
  • 19:52 jasmine@deploy1003: Finished scap sync-world: Deploying T427929 1360518 (duration: 10m 21s)
  • 19:47 jasmine@deploy1003: jasmine: Continuing with deployment
  • 19:46 jasmine@deploy1003: jasmine: Deploying T427929 1360518 synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 19:46 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/proton: apply
  • 19:43 jasmine@deploy1003: Started scap sync-world: Deploying T427929 1360518
  • 19:37 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/proton: apply
  • 19:36 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/proton: apply
  • 19:35 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/proton: apply
  • 19:35 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/proton: apply
  • 19:35 jforrester@deploy1003: helmfile [eqiad] DONE helmfile.d/services/proton: apply
  • 19:34 jforrester@deploy1003: helmfile [eqiad] START helmfile.d/services/proton: apply
  • 19:33 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/proton: apply
  • 19:32 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/proton: apply
  • 19:32 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/proton: apply
  • 19:32 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/proton: apply
  • 19:09 jasmine@deploy1003: Scap cancelled without rolling back.
  • 19:03 jasmine@deploy1003: jasmine: Deploying 1360512 T427929 synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 19:00 jasmine@deploy1003: Started scap sync-world: Deploying 1360512 T427929
  • 18:40 dduvall@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.48.0-wmf.1 refs T438223
  • 18:14 kamila@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
  • 18:13 kamila@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
  • 18:13 kamila@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
  • 18:12 kamila@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
  • 18:12 kamila@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
  • 18:11 kamila@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
  • 17:53 vgutierrez@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host lvs3010.esams.wmnet with OS trixie
  • 17:37 rzl@deploy1003: Finished scap sync-world: 1360004 T436338 (duration: 06m 40s)
  • 17:35 rzl@deploy1003: rzl: Continuing with deployment
  • 17:33 vgutierrez@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs3010.esams.wmnet with reason: host reimage
  • 17:32 rzl@deploy1003: rzl: 1360004 T436338 synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 17:31 rzl@deploy1003: Started scap sync-world: 1360004 T436338
  • 17:27 vgutierrez@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on lvs3010.esams.wmnet with reason: host reimage
  • 17:24 vgutierrez@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs3010.esams.wmnet (remove config only)
  • 17:24 vgutierrez@cumin1004: START - Cookbook sre.hosts.dhcp for host lvs3010.esams.wmnet (remove config only)
  • 17:22 jgreen@dns1004: END - running authdns-update
  • 17:20 jgreen@dns1004: START - running authdns-update
  • 17:17 cscott@deploy1003: Finished scap sync-world: Backport for ThumbExtractor: Only pass elements to Zest::matches() for captions (T441878) (1.47.0-wmf.24) (duration: 11m 16s)
  • 17:11 cscott@deploy1003: cscott: Continuing with deployment
  • 17:10 cscott@deploy1003: cscott: Backport for ThumbExtractor: Only pass elements to Zest::matches() for captions (T441878) (1.47.0-wmf.24) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 17:06 cscott@deploy1003: Started scap sync-world: Backport for ThumbExtractor: Only pass elements to Zest::matches() for captions (T441878) (1.47.0-wmf.24)
  • 17:05 pmiazga@deploy1003: mwscript-k8s job started: extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=kuwiki --logwiki=metawiki Bruhpedia Estoppel # T442867
  • 17:04 cscott@deploy1003: Finished scap sync-world: Backport for ThumbExtractor: Only pass elements to Zest::matches() for captions (T441878) (1.48.0-wmf.1) (duration: 14m 17s)
  • 17:03 vgutierrez@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs3010.esams.wmnet (push config only)
  • 17:03 vgutierrez@cumin1004: START - Cookbook sre.hosts.dhcp for host lvs3010.esams.wmnet (push config only)
  • 17:03 vgutierrez@cumin1004: START - Cookbook sre.hosts.reimage for host lvs3010.esams.wmnet with OS trixie
  • 16:58 cscott@deploy1003: cscott: Continuing with deployment
  • 16:54 cscott@deploy1003: cscott: Backport for ThumbExtractor: Only pass elements to Zest::matches() for captions (T441878) (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 16:49 cscott@deploy1003: Started scap sync-world: Backport for ThumbExtractor: Only pass elements to Zest::matches() for captions (T441878) (1.48.0-wmf.1)
  • 16:48 cooltey@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply
  • 16:47 cooltey@deploy1003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply
  • 16:47 cooltey@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply
  • 16:46 cooltey@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply
  • 16:46 cooltey@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply
  • 16:45 cooltey@deploy1003: helmfile [staging] START helmfile.d/services/wikifeeds: apply
  • 16:31 brouberol@deploy1003: Finished scap deploy-service airflow: Test (duration: 00m 07s)
  • 16:30 brouberol@deploy1003: scap deploy-service airflow: [PRODUCTION/CANARY] dse-k8s-eqiad Deploying airflow-test-k8s
  • 16:30 brouberol@deploy1003: Started scap deploy-service airflow: Test
  • 16:20 krinkle@deploy1003: Finished scap sync-world: Backport for MathML+MathJax rollout to phase 5 (Wikipedia small/medium/canary) (T271001) (duration: 13m 38s)
  • 16:15 krinkle@deploy1003: krinkle: Continuing with deployment
  • 16:11 krinkle@deploy1003: krinkle: Backport for MathML+MathJax rollout to phase 5 (Wikipedia small/medium/canary) (T271001) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 16:07 krinkle@deploy1003: Started scap sync-world: Backport for MathML+MathJax rollout to phase 5 (Wikipedia small/medium/canary) (T271001)
  • 16:01 ebernhardson@deploy1003: Finished scap sync-world: Backport for Reapply "cirrus: Send more_like traffic to eqiad" (duration: 13m 28s)
  • 15:57 klausman@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply
  • 15:57 klausman@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply
  • 15:56 klausman@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply
  • 15:55 klausman@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply
  • 15:54 klausman@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply
  • 15:54 klausman@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply
  • 15:53 ebernhardson@deploy1003: ebernhardson: Continuing with deployment
  • 15:53 ebernhardson@deploy1003: ebernhardson: Backport for Reapply "cirrus: Send more_like traffic to eqiad" synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 15:48 ebernhardson@deploy1003: Started scap sync-world: Backport for Reapply "cirrus: Send more_like traffic to eqiad"
  • 15:26 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P{lvs3009.esams.wmnet} and A:liberica
  • 15:25 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin config_reloading P{lvs3009.esams.wmnet} and A:liberica
  • 15:25 vgutierrez: pooling high-traffic1 services on high-traffic2 (lvs3009) in esams - T442859
  • 15:24 dancy@deploy1003: Installation of scap version "4.309.0" completed for 155 hosts
  • 15:20 dancy@deploy1003: Installing scap version "4.309.0" for 155 host(s)
  • 15:18 cgoubert@deploy1003: Finished scap sync-world: Backport for Fix Ib5079d9f9057b5a6bc66ddde6ba41606d3694775 (T440216) (duration: 33m 47s)
  • 15:12 brouberol@deploy1003: scap deploy-service airflow: [PRODUCTION/CANARY] dse-k8s-eqiad Deploying airflow-test-k8s
  • 15:12 brouberol@deploy1003: Started scap deploy-service airflow: Test
  • 15:09 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P{lvs3008.esams.wmnet} and A:liberica
  • 15:09 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin config_reloading P{lvs3008.esams.wmnet} and A:liberica
  • 15:09 vgutierrez: pooling upload/upload-https on high-traffic1 (lvs3008) in esams - T442859
  • 15:08 cmooney@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device cr2-eqiad
  • 15:08 cmooney@cumin1004: START - Cookbook sre.network.tls for network device cr2-eqiad
  • 15:05 cgoubert@deploy1003: cgoubert, tryvix1509: Continuing with deployment
  • 15:05 phuedx@deploy1003: Finished scap deploy-service eventgate-all: Updating all instances of EventGate to v1.35.0 (duration: 19m 07s)
  • 15:02 cmooney@cumin1004: END (FAIL) - Cookbook sre.network.tls (exit_code=99) for network device cr2-eqiad
  • 15:02 cmooney@cumin1004: START - Cookbook sre.network.tls for network device cr2-eqiad
  • 15:02 cgoubert@deploy1003: cgoubert, tryvix1509: Backport for Fix Ib5079d9f9057b5a6bc66ddde6ba41606d3694775 (T440216) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 15:01 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'.
  • 15:00 phuedx@deploy1003: scap deploy-service eventgate-all: [PRODUCTION/DEFAULT] codfw Deploying eventgate-analytics, eventgate-analytics-external, eventgate-logging-external, eventgate-main
  • 14:59 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'.
  • 14:58 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host build2004.codfw.wmnet
  • 14:54 phuedx@deploy1003: scap deploy-service eventgate-all: [PRODUCTION/DEFAULT] eqiad Deploying eventgate-analytics, eventgate-analytics-external, eventgate-logging-external, eventgate-main
  • 14:54 phuedx@deploy1003: scap deploy-service eventgate-all: [PRODUCTION/CANARY] codfw Deploying eventgate-analytics, eventgate-analytics-external, eventgate-logging-external, eventgate-main
  • 14:54 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host build2004.codfw.wmnet
  • 14:53 phuedx@deploy1003: scap deploy-service eventgate-all: [PRODUCTION/CANARY] eqiad Deploying eventgate-analytics, eventgate-analytics-external, eventgate-logging-external, eventgate-main
  • 14:53 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sretest1006.eqiad.wmnet
  • 14:52 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast5005.wikimedia.org
  • 14:47 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host sretest1006.eqiad.wmnet
  • 14:46 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast5005.wikimedia.org
  • 14:46 phuedx@deploy1003: scap deploy-service eventgate-all: [STAGING/DEFAULT] staging-eqiad Deploying eventgate-analytics, eventgate-analytics-external, eventgate-logging-external, eventgate-main
  • 14:46 phuedx@deploy1003: Started scap deploy-service eventgate-all: Updating all instances of EventGate to v1.35.0
  • 14:44 cgoubert@deploy1003: Started scap sync-world: Backport for Fix Ib5079d9f9057b5a6bc66ddde6ba41606d3694775 (T440216)
  • 14:44 cmooney@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device cr2-eqsin
  • 14:43 cmooney@cumin1004: START - Cookbook sre.network.tls for network device cr2-eqsin
  • 14:43 cmooney@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device ssw1-e1-eqiad
  • 14:43 cmooney@cumin1004: START - Cookbook sre.network.tls for network device ssw1-e1-eqiad
  • 14:43 cmooney@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-a3-codfw
  • 14:43 cmooney@cumin1004: START - Cookbook sre.network.tls for network device lsw1-a3-codfw
  • 14:43 cmooney@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-a8-codfw
  • 14:43 cmooney@cumin1004: START - Cookbook sre.network.tls for network device lsw1-a8-codfw
  • 14:43 cmooney@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-a7-codfw
  • 14:43 cmooney@cumin1004: START - Cookbook sre.network.tls for network device lsw1-a7-codfw
  • 14:42 cmooney@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-b3-codfw
  • 14:42 cmooney@cumin1004: START - Cookbook sre.network.tls for network device lsw1-b3-codfw
  • 14:42 cmooney@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-b4-codfw
  • 14:42 cmooney@cumin1004: START - Cookbook sre.network.tls for network device lsw1-b4-codfw
  • 14:42 cmooney@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-b7-codfw
  • 14:42 cmooney@cumin1004: START - Cookbook sre.network.tls for network device lsw1-b7-codfw
  • 14:42 cmooney@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device ssw1-a1-codfw
  • 14:42 cmooney@cumin1004: START - Cookbook sre.network.tls for network device ssw1-a1-codfw
  • 14:42 cmooney@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device ssw1-a8-codfw
  • 14:41 cmooney@cumin1004: START - Cookbook sre.network.tls for network device ssw1-a8-codfw
  • 14:39 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync
  • 14:38 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync
  • 14:33 hashar: Upgrading CI Jenkins
  • 14:28 Tran: Deployed patch for T441775
  • 14:15 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'.
  • 14:13 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'.
  • 14:06 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host urldownloader1006.wikimedia.org
  • 14:04 moritzm: installing libpng1.6 security updates
  • 14:03 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host apt1002.wikimedia.org
  • 14:03 stran@deploy1003: Finished scap sync-world: Backport for Enable SuggestedInvestigations case generation on ptwiki (T442582) (duration: 12m 53s)
  • 14:03 kamila@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
  • 14:02 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host urldownloader1006.wikimedia.org
  • 14:00 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host urldownloader1005.wikimedia.org
  • 13:59 hashar: Upgrading and restarting Jenkins on releases1003 | https://releases-jenkins.wikimedia.org/
  • 13:57 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host apt1002.wikimedia.org
  • 13:57 stran@deploy1003: stran: Continuing with deployment
  • 13:56 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host urldownloader1005.wikimedia.org
  • 13:56 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-rw2001.wikimedia.org
  • 13:55 kamila@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
  • 13:55 kamila@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
  • 13:55 stran@deploy1003: stran: Backport for Enable SuggestedInvestigations case generation on ptwiki (T442582) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 13:55 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host rpki2003.codfw.wmnet with OS trixie
  • 13:53 kamila@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
  • 13:52 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ldap-rw2001.wikimedia.org
  • 13:50 stran@deploy1003: Started scap sync-world: Backport for Enable SuggestedInvestigations case generation on ptwiki (T442582)
  • 13:45 samtar@deploy1003: Finished scap sync-world: Backport for [arwiktionary] Add "وق" as a project namespace alias (T442847) (duration: 11m 40s)
  • 13:39 samtar@deploy1003: gergesshamon, samtar: Continuing with deployment
  • 13:39 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-rw1001.wikimedia.org
  • 13:38 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on rpki2003.codfw.wmnet with reason: host reimage
  • 13:38 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ldap-rw1001.wikimedia.org
  • 13:37 samtar@deploy1003: gergesshamon, samtar: Backport for [arwiktionary] Add "وق" as a project namespace alias (T442847) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 13:33 samtar@deploy1003: Started scap sync-world: Backport for [arwiktionary] Add "وق" as a project namespace alias (T442847)
  • 13:32 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on rpki2003.codfw.wmnet with reason: host reimage
  • 13:30 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host rpki2003.codfw.wmnet (remove config only)
  • 13:30 jmm@cumin2003: START - Cookbook sre.hosts.dhcp for host rpki2003.codfw.wmnet (remove config only)
  • 13:25 samtar@deploy1003: Finished scap sync-world: Backport for Require 5 edits for autoconfirmed users on English Wikiversity (T442057) (duration: 12m 43s)
  • 13:25 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aux-k8s-etcd1004.eqiad.wmnet
  • 13:24 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host urldownloader2006.wikimedia.org
  • 13:21 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host aux-k8s-etcd1004.eqiad.wmnet
  • 13:20 gkyziridis@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' .
  • 13:20 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revertrisk' for release 'main' .
  • 13:19 samtar@deploy1003: samtar, codenamenoreste: Continuing with deployment
  • 13:19 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host urldownloader2006.wikimedia.org
  • 13:17 samtar@deploy1003: samtar, codenamenoreste: Backport for Require 5 edits for autoconfirmed users on English Wikiversity (T442057) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 13:17 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host rpki2003.codfw.wmnet (push config only)
  • 13:16 jmm@cumin2003: START - Cookbook sre.hosts.dhcp for host rpki2003.codfw.wmnet (push config only)
  • 13:16 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P{cp[6012,6016].drmrs.wmnet} and A:cp
  • 13:16 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp6016.drmrs.wmnet
  • 13:15 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host rpki2003.codfw.wmnet with OS trixie
  • 13:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host urldownloader2005.wikimedia.org
  • 13:13 samtar@deploy1003: Started scap sync-world: Backport for Require 5 edits for autoconfirmed users on English Wikiversity (T442057)
  • 13:10 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host urldownloader2005.wikimedia.org
  • 13:05 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aux-k8s-etcd1005.eqiad.wmnet
  • 13:02 root@cumin2003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-master-eqiad
  • 13:02 root@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1003.eqiad.wmnet
  • 13:02 root@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1003.eqiad.wmnet
  • 13:01 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host aux-k8s-etcd1005.eqiad.wmnet
  • 12:57 root@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1003.eqiad.wmnet
  • 12:57 root@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1003.eqiad.wmnet
  • 12:57 root@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1002.eqiad.wmnet
  • 12:57 root@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1002.eqiad.wmnet
  • 12:53 reedy@deploy1003: Finished scap sync-world: Backport for InitialiseSettings: Remove $wgOATHAuth2FAForAllWarnings (duration: 15m 26s)
  • 12:52 root@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1002.eqiad.wmnet
  • 12:52 root@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1002.eqiad.wmnet
  • 12:52 root@cumin2003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-eqiad
  • 12:51 moritzm: installing NSS security updates
  • 12:51 root@cumin2003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-master-codfw
  • 12:51 root@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl2003.codfw.wmnet
  • 12:51 root@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl2003.codfw.wmnet
  • 12:48 reedy@deploy1003: reedy: Continuing with deployment
  • 12:46 root@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl2003.codfw.wmnet
  • 12:46 root@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl2003.codfw.wmnet
  • 12:46 root@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl2002.codfw.wmnet
  • 12:46 root@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl2002.codfw.wmnet
  • 12:42 reedy@deploy1003: reedy: Backport for InitialiseSettings: Remove $wgOATHAuth2FAForAllWarnings synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 12:41 root@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl2002.codfw.wmnet
  • 12:41 root@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl2002.codfw.wmnet
  • 12:41 root@cumin2003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-codfw
  • 12:41 vgutierrez@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host lvs3010.esams.wmnet with OS trixie
  • 12:38 reedy@deploy1003: Started scap sync-world: Backport for InitialiseSettings: Remove $wgOATHAuth2FAForAllWarnings
  • 12:35 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp6012.drmrs.wmnet
  • 12:32 dbrant@deploy1003: helmfile [codfw] DONE helmfile.d/services/push-notifications: apply
  • 12:31 dbrant@deploy1003: helmfile [codfw] START helmfile.d/services/push-notifications: apply
  • 12:31 dbrant@deploy1003: helmfile [eqiad] DONE helmfile.d/services/push-notifications: apply
  • 12:30 dbrant@deploy1003: helmfile [eqiad] START helmfile.d/services/push-notifications: apply
  • 12:30 vgutierrez@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs3010.esams.wmnet (push config only)
  • 12:30 vgutierrez@cumin1004: START - Cookbook sre.hosts.dhcp for host lvs3010.esams.wmnet (push config only)
  • 12:30 vgutierrez@cumin1004: START - Cookbook sre.hosts.reimage for host lvs3010.esams.wmnet with OS trixie
  • 12:29 vgutierrez@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host lvs3010.esams.wmnet with OS trixie
  • 12:29 dbrant@deploy1003: helmfile [staging] DONE helmfile.d/services/push-notifications: apply
  • 12:29 dbrant@deploy1003: helmfile [staging] START helmfile.d/services/push-notifications: apply
  • 12:24 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-reboot rolling reboot on P{cp[6012,6016].drmrs.wmnet} and A:cp
  • 12:22 ladsgroup@deploy1003: Finished scap sync-world: Backport for Upgrading symfony/mailer and symfony/mime (v7.4.17 => v7.4.22) (1.47.0-wmf.24), ParserOutputAccess: Don't load dirty entry if not needed (T442669) (1.47.0-wmf.24) (duration: 13m 00s)
  • 12:17 ladsgroup@deploy1003: ladsgroup, ladsgroup-claude: Continuing with deployment
  • 12:14 ladsgroup@deploy1003: ladsgroup, ladsgroup-claude: Backport for Upgrading symfony/mailer and symfony/mime (v7.4.17 => v7.4.22) (1.47.0-wmf.24), ParserOutputAccess: Don't load dirty entry if not needed (T442669) (1.47.0-wmf.24) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 12:09 ladsgroup@deploy1003: Started scap sync-world: Backport for Upgrading symfony/mailer and symfony/mime (v7.4.17 => v7.4.22) (1.47.0-wmf.24), ParserOutputAccess: Don't load dirty entry if not needed (T442669) (1.47.0-wmf.24)
  • 12:04 vgutierrez@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs3010.esams.wmnet (push config only)
  • 12:04 vgutierrez@cumin1004: START - Cookbook sre.hosts.dhcp for host lvs3010.esams.wmnet (push config only)
  • 12:03 vgutierrez@cumin1004: START - Cookbook sre.hosts.reimage for host lvs3010.esams.wmnet with OS trixie
  • 12:02 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P{lvs6001.*} and A:liberica
  • 12:02 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin config_reloading P{lvs6001.*} and A:liberica
  • 12:00 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
  • 12:00 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
  • 12:00 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
  • 12:00 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: sync
  • 11:57 gkyziridis@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'edit-check' for release 'main' .
  • 11:56 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'edit-check' for release 'main' .
  • 11:53 gkyziridis@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'edit-check' for release 'main' .
  • 11:47 vgutierrez@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host lvs6001.drmrs.wmnet with OS trixie
  • 11:33 ladsgroup@deploy1003: Finished scap sync-world: Backport for Upgrading symfony/mailer and symfony/mime (v7.4.19 => v7.4.22) (T442768) (1.48.0-wmf.1), ParserOutputAccess: Don't load dirty entry if not needed (T442669) (1.48.0-wmf.1) (duration: 14m 35s)
  • 11:29 vgutierrez@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs6001.drmrs.wmnet with reason: host reimage
  • 11:27 ladsgroup@deploy1003: ladsgroup: Continuing with deployment
  • 11:23 ladsgroup@deploy1003: ladsgroup: Backport for Upgrading symfony/mailer and symfony/mime (v7.4.19 => v7.4.22) (T442768) (1.48.0-wmf.1), ParserOutputAccess: Don't load dirty entry if not needed (T442669) (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 11:23 vgutierrez@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on lvs6001.drmrs.wmnet with reason: host reimage
  • 11:22 cwilliams@cumin1004: END (PASS) - Cookbook sre.backup.offsite (exit_code=0)
  • 11:22 cwilliams@cumin1004: START - Cookbook sre.backup.offsite
  • 11:20 vgutierrez@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs6001.drmrs.wmnet (remove config only)
  • 11:20 vgutierrez@cumin1004: START - Cookbook sre.hosts.dhcp for host lvs6001.drmrs.wmnet (remove config only)
  • 11:20 vgutierrez: upload purged 0.26 to trixie-wikimedia (apt.wm.o ) - T442736
  • 11:18 ladsgroup@deploy1003: Started scap sync-world: Backport for Upgrading symfony/mailer and symfony/mime (v7.4.19 => v7.4.22) (T442768) (1.48.0-wmf.1), ParserOutputAccess: Don't load dirty entry if not needed (T442669) (1.48.0-wmf.1)
  • 11:07 vgutierrez@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs6001.drmrs.wmnet (push config only)
  • 11:07 vgutierrez@cumin1004: START - Cookbook sre.hosts.dhcp for host lvs6001.drmrs.wmnet (push config only)
  • 11:06 vgutierrez@cumin1004: START - Cookbook sre.hosts.reimage for host lvs6001.drmrs.wmnet with OS trixie
  • 10:59 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P{lvs6001.*} and A:liberica
  • 10:59 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin config_reloading P{lvs6001.*} and A:liberica
  • 10:49 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P{lvs6002.*} and A:liberica
  • 10:49 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin config_reloading P{lvs6002.*} and A:liberica
  • 10:48 blake@deploy1003: Finished scap sync-world: Enable non-batch k8s native sidecars (T417800) (duration: 07m 55s)
  • 10:41 blake@deploy1003: Started scap sync-world: Enable non-batch k8s native sidecars (T417800)
  • 10:40 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host apt-staging2001.codfw.wmnet
  • 10:40 blake@deploy1003: Failed scap sync-world: Enable non-batch k8s native sidecars (T417800) (duration: 14m 32s)
  • 10:36 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host apt-staging2001.codfw.wmnet
  • 10:34 moritzm: installing Apache httpd security updates
  • 10:34 blake@deploy1003: Rolling back deployment
  • 10:27 blake@deploy1003: Started scap sync-world: Enable non-batch k8s native sidecars (T417800)
  • 10:26 vgutierrez@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host lvs6002.drmrs.wmnet with OS trixie
  • 10:24 moritzm: installing Unbound security updates
  • 10:13 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' .
  • 10:12 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' .
  • 10:11 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' .
  • 10:08 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' .
  • 10:07 vgutierrez@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs6002.drmrs.wmnet with reason: host reimage
  • 10:05 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' .
  • 10:04 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' .
  • 10:04 vgutierrez@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on lvs6002.drmrs.wmnet with reason: host reimage
  • 10:01 vgutierrez@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs6002.drmrs.wmnet (remove config only)
  • 10:01 vgutierrez@cumin1004: START - Cookbook sre.hosts.dhcp for host lvs6002.drmrs.wmnet (remove config only)
  • 09:59 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' .
  • 09:53 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' .
  • 09:52 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' .
  • 09:49 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' .
  • 09:48 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' .
  • 09:47 vgutierrez@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs6002.drmrs.wmnet (push config only)
  • 09:47 vgutierrez@cumin1004: START - Cookbook sre.hosts.dhcp for host lvs6002.drmrs.wmnet (push config only)
  • 09:46 vgutierrez@cumin1004: START - Cookbook sre.hosts.reimage for host lvs6002.drmrs.wmnet with OS trixie
  • 09:45 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' .
  • 09:41 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' .
  • 09:41 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P{lvs6002.*} and A:liberica
  • 09:40 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin config_reloading P{lvs6002.*} and A:liberica
  • 09:38 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' .
  • 09:37 moritzm: installing Bind security updates
  • 09:36 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' .
  • 09:34 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-articletopic' for release 'main' .
  • 09:33 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-articletopic' for release 'main' .
  • 09:33 dpogorzelski@cumin1004: conftool action : set/pooled=true; selector: dnsdisc=k8s-ingress-ml-serve,name=eqiad
  • 09:32 dpogorzelski@cumin1004: conftool action : set/pooled=true; selector: dnsdisc=inference,name=eqiad
  • 09:32 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-articletopic' for release 'main' .
  • 09:28 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' .
  • 09:28 jiji@cumin2003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1228: Depooling replica - T442815
  • 09:27 jiji@cumin2003: START - Cookbook sre.mysql.depool depool db1228: Depooling replica - T442815
  • 09:25 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' .
  • 09:23 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' .
  • 09:17 moritzm: installing ca-certificates updates on Bookworm
  • 09:16 vgutierrez@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host lvs6003.drmrs.wmnet with OS trixie
  • 08:56 vgutierrez@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs6003.drmrs.wmnet with reason: host reimage
  • 08:51 vgutierrez@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on lvs6003.drmrs.wmnet with reason: host reimage
  • 08:49 vgutierrez@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs6003.drmrs.wmnet (remove config only)
  • 08:48 vgutierrez@cumin1004: START - Cookbook sre.hosts.dhcp for host lvs6003.drmrs.wmnet (remove config only)
  • 08:42 hashar: restarting Gerrit
  • 08:38 jelto@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:15:00 on 14 hosts with reason: Java upgrade
  • 08:38 jelto@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:15:00 on gerrit1003.wikimedia.org with reason: Java upgrade
  • 08:35 vgutierrez@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs6003.drmrs.wmnet (push config only)
  • 08:35 vgutierrez@cumin1004: START - Cookbook sre.hosts.dhcp for host lvs6003.drmrs.wmnet (push config only)
  • 08:34 vgutierrez@cumin1004: START - Cookbook sre.hosts.reimage for host lvs6003.drmrs.wmnet with OS trixie
  • 08:30 hashar: Restarted Gerrit on gerrit2002/gerrit2003 (spare/replica) to switch to Java 21 | T392465
  • 08:30 moritzm: imported Jenkins 2.580.2 to thirdparty/jenkins for trixie-wikimedia T442030
  • 07:43 kevinbazira@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
  • 07:43 kevinbazira@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
  • 06:56 marostegui: DB circular replication has been disabled everywhere T436501
  • 06:56 marostegui@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from eqiad to codfw for section s1
  • 06:55 marostegui@cumin1004: START - Cookbook sre.switchdc.databases.finalize for the switch from eqiad to codfw for section s1
  • 06:53 marostegui@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from eqiad to codfw for section s4
  • 06:53 marostegui@cumin1004: START - Cookbook sre.switchdc.databases.finalize for the switch from eqiad to codfw for section s4
  • 06:53 marostegui@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from eqiad to codfw for section s8
  • 06:53 marostegui@cumin1004: START - Cookbook sre.switchdc.databases.finalize for the switch from eqiad to codfw for section s8
  • 06:52 marostegui@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from eqiad to codfw for section s7
  • 06:52 marostegui@cumin1004: START - Cookbook sre.switchdc.databases.finalize for the switch from eqiad to codfw for section s7
  • 06:52 marostegui@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from eqiad to codfw for section s3
  • 06:51 marostegui@cumin1004: START - Cookbook sre.switchdc.databases.finalize for the switch from eqiad to codfw for section s3
  • 06:51 marostegui@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from eqiad to codfw for section s2
  • 06:50 marostegui@cumin1004: START - Cookbook sre.switchdc.databases.finalize for the switch from eqiad to codfw for section s2
  • 06:49 marostegui@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from eqiad to codfw for section s5
  • 06:49 marostegui@cumin1004: START - Cookbook sre.switchdc.databases.finalize for the switch from eqiad to codfw for section s5
  • 06:47 marostegui@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from eqiad to codfw for section s6
  • 06:47 marostegui@cumin1004: START - Cookbook sre.switchdc.databases.finalize for the switch from eqiad to codfw for section s6
  • 06:46 marostegui@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from eqiad to codfw for section es7
  • 06:46 marostegui@cumin1004: START - Cookbook sre.switchdc.databases.finalize for the switch from eqiad to codfw for section es7
  • 06:43 marostegui@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from eqiad to codfw for section es6
  • 06:43 marostegui@cumin1004: START - Cookbook sre.switchdc.databases.finalize for the switch from eqiad to codfw for section es6
  • 06:40 marostegui@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from eqiad to codfw for section x4
  • 06:40 marostegui@cumin1004: START - Cookbook sre.switchdc.databases.finalize for the switch from eqiad to codfw for section x4
  • 06:38 marostegui@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from eqiad to codfw for section x3
  • 06:37 marostegui@cumin1004: START - Cookbook sre.switchdc.databases.finalize for the switch from eqiad to codfw for section x3
  • 06:36 marostegui@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from eqiad to codfw for section x1
  • 06:35 marostegui@cumin1004: START - Cookbook sre.switchdc.databases.finalize for the switch from eqiad to codfw for section x1
  • 06:10 kartik@deploy1003: helmfile [eqiad] DONE helmfile.d/services/machinetranslation: apply
  • 06:03 kartik@deploy1003: helmfile [eqiad] START helmfile.d/services/machinetranslation: apply
  • 06:00 kevinbazira@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
  • 05:54 kartik@deploy1003: helmfile [codfw] DONE helmfile.d/services/machinetranslation: apply
  • 05:49 kartik@deploy1003: helmfile [codfw] START helmfile.d/services/machinetranslation: apply
  • 05:48 kartik@deploy1003: helmfile [staging] DONE helmfile.d/services/machinetranslation: apply
  • 05:48 kevinbazira@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
  • 05:48 kevinbazira@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
  • 05:45 kartik@deploy1003: helmfile [staging] START helmfile.d/services/machinetranslation: apply
  • 05:27 kartik@deploy1003: helmfile [ml-serve-codfw] 'sync' command on namespace 'recommendation-api-ng' for release 'main' .
  • 05:23 kartik@deploy1003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'recommendation-api-ng' for release 'main' .
  • 05:20 kartik@deploy1003: helmfile [ml-staging-codfw] 'sync' command on namespace 'recommendation-api-ng' for release 'main' .
  • 04:14 herron@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1262: depooling replica after host flap
  • 04:11 herron@cumin1004: START - Cookbook sre.mysql.depool depool db1262: depooling replica after host flap
  • 00:00 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
  • 00:00 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: sync

2026-10-28

  • 23:12 egardner@deploy1003: Finished scap sync-world: Backport for ShareCard: Show page views as a monthly rate (T439164) (1.48.0-wmf.1), ShareCard: Hide images on the bad image list (T439161) (1.48.0-wmf.1), Bump symfony/mailer and symfony/mime to 7.4.22 (T442768) (1.47.0-wmf.24, 1.48.0-wmf.1) (duration: 36m 58s)
  • 23:00 egardner@deploy1003: sbassett, egardner: Continuing with deployment
  • 22:56 egardner@deploy1003: sbassett, egardner: Backport for ShareCard: Show page views as a monthly rate (T439164) (1.48.0-wmf.1), ShareCard: Hide images on the bad image list (T439161) (1.48.0-wmf.1), Bump symfony/mailer and symfony/mime to 7.4.22 (T442768) (1.47.0-wmf.24, 1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/
  • 22:35 egardner@deploy1003: Started scap sync-world: Backport for ShareCard: Show page views as a monthly rate (T439164) (1.48.0-wmf.1), ShareCard: Hide images on the bad image list (T439161) (1.48.0-wmf.1), Bump symfony/mailer and symfony/mime to 7.4.22 (T442768) (1.47.0-wmf.24, 1.48.0-wmf.1)
  • 22:30 egardner@deploy1003: Finished scap sync-world: Backport for Disable Discord QuickSurvey (duration: 10m 41s)
  • 22:25 egardner@deploy1003: derenrich, egardner: Continuing with deployment
  • 22:24 egardner@deploy1003: derenrich, egardner: Backport for Disable Discord QuickSurvey synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 22:20 egardner@deploy1003: Started scap sync-world: Backport for Disable Discord QuickSurvey
  • 22:17 jdlrobson@deploy1003: Finished scap sync-world: Backport for Enable Vector ReadingList CTA on group 1 wikipedias+mw+wikitech (T441705) (duration: 12m 13s)
  • 22:11 jdlrobson@deploy1003: jdlrobson: Continuing with deployment
  • 22:09 jdlrobson@deploy1003: jdlrobson: Backport for Enable Vector ReadingList CTA on group 1 wikipedias+mw+wikitech (T441705) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 22:05 jdlrobson@deploy1003: Started scap sync-world: Backport for Enable Vector ReadingList CTA on group 1 wikipedias+mw+wikitech (T441705)
  • 21:51 ammarpad@deploy1003: mwscript-k8s job started: extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=wikimania2007wiki --logwiki=metawiki MogosCZ Mogoskii # T442713
  • 21:47 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-worker-eqiad
  • 21:47 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker-exp1001.eqiad.wmnet
  • 21:46 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker-exp1001.eqiad.wmnet
  • 21:42 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker-exp1001.eqiad.wmnet
  • 21:42 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker-exp1001.eqiad.wmnet
  • 21:42 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1371-1384].eqiad.wmnet
  • 21:42 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1371-1384].eqiad.wmnet
  • 21:34 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1371-1384].eqiad.wmnet
  • 21:33 mutante: deploy1003 - sudo find . -path '*/.git/DEPLOY_HEAD' -exec sed -i -E '/^git_(server|scheme|deploy_dir):/d' {} + (backup in /root/DEPLOY_HEAD-backup.tgz) - T197470
  • 21:26 pt1979@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host ml-serve1016.eqiad.wmnet (live debug)
  • 21:25 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1371-1384].eqiad.wmnet
  • 21:25 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1357-1370].eqiad.wmnet
  • 21:24 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1357-1370].eqiad.wmnet
  • 21:22 akhatun@deploy1003: helmfile [codfw] DONE helmfile.d/services/lac-lambdas: apply
  • 21:21 akhatun@deploy1003: helmfile [codfw] START helmfile.d/services/lac-lambdas: apply
  • 21:21 akhatun@deploy1003: helmfile [eqiad] DONE helmfile.d/services/lac-lambdas: apply
  • 21:21 akhatun@deploy1003: helmfile [eqiad] START helmfile.d/services/lac-lambdas: apply
  • 21:20 akhatun@deploy1003: helmfile [staging] DONE helmfile.d/services/lac-lambdas: apply
  • 21:20 akhatun@deploy1003: helmfile [staging] START helmfile.d/services/lac-lambdas: apply
  • 21:19 akhatun@deploy1003: helmfile [staging] DONE helmfile.d/services/lac-lambdas: apply
  • 21:18 akhatun@deploy1003: helmfile [staging] START helmfile.d/services/lac-lambdas: apply
  • 21:16 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1357-1370].eqiad.wmnet
  • 21:14 jforrester@deploy1003: Finished scap deploy-service wikifunctions: Weekly f-o service deploy (duration: 01m 47s)
  • 21:14 jforrester@deploy1003: scap deploy-service wikifunctions: [PRODUCTION/DEFAULT] codfw Deploying wikifunctions
  • 21:13 jforrester@deploy1003: scap deploy-service wikifunctions: [PRODUCTION/DEFAULT] eqiad Deploying wikifunctions
  • 21:12 jforrester@deploy1003: scap deploy-service wikifunctions: [STAGING/DEFAULT] staging-eqiad Deploying wikifunctions
  • 21:12 jforrester@deploy1003: Started scap deploy-service wikifunctions: Weekly f-o service deploy
  • 21:08 jforrester@deploy1003: Finished scap deploy-service wikifunctions: Weekly f-e service deploy, this time with a bump commit (duration: 02m 11s)
  • 21:07 jforrester@deploy1003: scap deploy-service wikifunctions: [PRODUCTION/DEFAULT] codfw Deploying wikifunctions
  • 21:07 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1357-1370].eqiad.wmnet
  • 21:07 jforrester@deploy1003: scap deploy-service wikifunctions: [PRODUCTION/DEFAULT] eqiad Deploying wikifunctions
  • 21:06 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1343-1356].eqiad.wmnet
  • 21:06 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1343-1356].eqiad.wmnet
  • 21:06 jforrester@deploy1003: scap deploy-service wikifunctions: [STAGING/DEFAULT] staging-eqiad Deploying wikifunctions
  • 21:06 jforrester@deploy1003: Started scap deploy-service wikifunctions: Weekly f-e service deploy, this time with a bump commit
  • 21:02 jforrester@deploy1003: Finished scap deploy-service wikifunctions: Weekly function-evaluator service deploy (duration: 01m 29s)
  • 21:01 jforrester@deploy1003: scap deploy-service wikifunctions: [PRODUCTION/DEFAULT] codfw Deploying wikifunctions
  • 21:01 jforrester@deploy1003: scap deploy-service wikifunctions: [PRODUCTION/DEFAULT] eqiad Deploying wikifunctions
  • 21:00 jforrester@deploy1003: scap deploy-service wikifunctions: [STAGING/DEFAULT] staging-eqiad Deploying wikifunctions
  • 21:00 jforrester@deploy1003: Started scap deploy-service wikifunctions: Weekly function-evaluator service deploy
  • 20:59 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1343-1356].eqiad.wmnet
  • 20:50 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1343-1356].eqiad.wmnet
  • 20:50 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1329-1342].eqiad.wmnet
  • 20:50 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1329-1342].eqiad.wmnet
  • 20:42 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1329-1342].eqiad.wmnet
  • 20:40 sbassett: Deployed security mitigations for T435455, T439844, T442538
  • 20:29 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1329-1342].eqiad.wmnet
  • 20:29 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1315-1328].eqiad.wmnet
  • 20:29 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1315-1328].eqiad.wmnet
  • 20:20 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1315-1328].eqiad.wmnet
  • 20:19 mutante: etherpad1004 - apache httpd disabled by puppet - etherpad running normally
  • 20:17 arlolra@deploy1003: Finished scap sync-world: Backport for Turn off "report a visual" bug on wikipedias (T430970) (duration: 11m 26s)
  • 20:14 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 20:14 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update last few IPs that got awawy - cmooney@cumin1004"
  • 20:14 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update last few IPs that got awawy - cmooney@cumin1004"
  • 20:12 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1315-1328].eqiad.wmnet
  • 20:12 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1301-1314].eqiad.wmnet
  • 20:12 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1301-1314].eqiad.wmnet
  • 20:11 arlolra@deploy1003: arlolra: Continuing with deployment
  • 20:10 arlolra@deploy1003: arlolra: Backport for Turn off "report a visual" bug on wikipedias (T430970) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 20:10 cmooney@cumin1004: START - Cookbook sre.dns.netbox
  • 20:06 arlolra@deploy1003: Started scap sync-world: Backport for Turn off "report a visual" bug on wikipedias (T430970)
  • 20:03 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1301-1314].eqiad.wmnet
  • 20:03 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 20:03 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update subint entries for row B interfaces on eqiad CRs - cmooney@cumin1004"
  • 20:03 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update subint entries for row B interfaces on eqiad CRs - cmooney@cumin1004"
  • 19:59 cmooney@cumin1004: START - Cookbook sre.dns.netbox
  • 19:55 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1301-1314].eqiad.wmnet
  • 19:54 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1284-1289,1291-1297,1299].eqiad.wmnet
  • 19:54 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1284-1289,1291-1297,1299].eqiad.wmnet
  • 19:54 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 19:54 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update subint entries for row B interfaces on eqiad CRs - cmooney@cumin1004"
  • 19:54 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update subint entries for row B interfaces on eqiad CRs - cmooney@cumin1004"
  • 19:52 akhatun@deploy1003: Finished scap sync-world: Backport for Prefetch the LAC revision_feature_counts artifact on group1 wikipedias (duration: 11m 48s)
  • 19:52 pt1979@cumin1004: START - Cookbook sre.hosts.dhcp for host ml-serve1016.eqiad.wmnet (live debug)
  • 19:49 cmooney@cumin1004: START - Cookbook sre.dns.netbox
  • 19:46 akhatun@deploy1003: akhatun, otto: Continuing with deployment
  • 19:45 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1284-1289,1291-1297,1299].eqiad.wmnet
  • 19:45 akhatun@deploy1003: akhatun, otto: Backport for Prefetch the LAC revision_feature_counts artifact on group1 wikipedias synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 19:40 akhatun@deploy1003: Started scap sync-world: Backport for Prefetch the LAC revision_feature_counts artifact on group1 wikipedias
  • 19:37 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1284-1289,1291-1297,1299].eqiad.wmnet
  • 19:37 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1269-1283].eqiad.wmnet
  • 19:37 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1269-1283].eqiad.wmnet
  • 19:30 akhatun@deploy1003: helmfile [codfw] DONE helmfile.d/services/lac-lambdas: apply
  • 19:29 akhatun@deploy1003: helmfile [codfw] START helmfile.d/services/lac-lambdas: apply
  • 19:28 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1269-1283].eqiad.wmnet
  • 19:26 akhatun@deploy1003: helmfile [eqiad] DONE helmfile.d/services/lac-lambdas: apply
  • 19:26 akhatun@deploy1003: helmfile [eqiad] START helmfile.d/services/lac-lambdas: apply
  • 19:23 akhatun@deploy1003: helmfile [staging] DONE helmfile.d/services/lac-lambdas: apply
  • 19:22 akhatun@deploy1003: helmfile [staging] START helmfile.d/services/lac-lambdas: apply
  • 19:21 inflatador: [bking@deploy1003] `curl -XPOST -u $U -s https://opensearch-ttmserver.svc.eqiad.wmnet:30443/_cluster/reroute` Fix yellow status
  • 19:18 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1269-1283].eqiad.wmnet
  • 19:18 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1254-1268].eqiad.wmnet
  • 19:18 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1254-1268].eqiad.wmnet
  • 19:09 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1254-1268].eqiad.wmnet
  • 18:59 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1254-1268].eqiad.wmnet
  • 18:59 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1165,1240-1253].eqiad.wmnet
  • 18:59 dancy@deploy1003: Installation of scap version "4.307.0" completed for 2 hosts
  • 18:59 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1165,1240-1253].eqiad.wmnet
  • 18:58 dancy@deploy1003: Installing scap version "4.307.0" for 2 host(s)
  • 18:57 Dreamy_Jazz: Running `foreachwikiindblist group1.dblist cleanupWatchlistTemporaryAccountRows.php` for T439059
  • 18:50 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1165,1240-1253].eqiad.wmnet
  • 18:45 dancy@deploy1003: Installing scap version "4.307.0" for 155 host(s)
  • 18:41 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1165,1240-1253].eqiad.wmnet
  • 18:41 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1149-1151,1153-1164].eqiad.wmnet
  • 18:41 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1149-1151,1153-1164].eqiad.wmnet
  • 18:32 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1149-1151,1153-1164].eqiad.wmnet
  • 18:31 rzl: rzl@apt1002:~$ sudo -i reprepro copy bullseye-wikimedia bookworm-wikimedia envoyproxy # backwards, intended
  • 18:31 rzl: rzl@apt1002:~$ sudo -i reprepro copy trixie-wikimedia bookworm-wikimedia envoyproxy
  • 18:30 rzl: rzl@apt1002:~$ sudo -i reprepro -C main includedeb bookworm-wikimedia /srv/wikimedia/pool/component/envoy-future/e/envoyproxy/envoyproxy_1.40.1-1_amd64.deb
  • 18:26 tchin@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventstreams: apply
  • 18:25 dduvall@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.48.0-wmf.1 refs T438223
  • 18:25 tchin@deploy1003: helmfile [codfw] START helmfile.d/services/eventstreams: apply
  • 18:25 tchin@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventstreams: apply
  • 18:19 tchin@deploy1003: helmfile [codfw] START helmfile.d/services/eventstreams: apply
  • 18:19 rzl@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply
  • 18:19 rzl@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply
  • 18:19 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1149-1151,1153-1164].eqiad.wmnet
  • 18:18 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1134-1148].eqiad.wmnet
  • 18:18 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1134-1148].eqiad.wmnet
  • 18:17 tchin@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventstreams: apply
  • 18:10 tchin@deploy1003: helmfile [codfw] START helmfile.d/services/eventstreams: apply
  • 18:10 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1134-1148].eqiad.wmnet
  • 18:07 tchin@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventstreams: apply
  • 18:02 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1134-1148].eqiad.wmnet
  • 18:01 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1119-1133].eqiad.wmnet
  • 18:01 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1119-1133].eqiad.wmnet
  • 17:56 tchin@deploy1003: helmfile [codfw] START helmfile.d/services/eventstreams: apply
  • 17:52 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1119-1133].eqiad.wmnet
  • 17:45 aqu@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply
  • 17:45 aqu@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply
  • 17:43 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1119-1133].eqiad.wmnet
  • 17:43 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1080-1081,1084-1087,1093-1095,1113-1118].eqiad.wmnet
  • 17:43 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1080-1081,1084-1087,1093-1095,1113-1118].eqiad.wmnet
  • 17:35 rzl@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply
  • 17:35 rzl@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply
  • 17:34 rzl@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-syntaxhighlight: apply
  • 17:33 rzl@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-syntaxhighlight: apply
  • 17:32 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1080-1081,1084-1087,1093-1095,1113-1118].eqiad.wmnet
  • 17:29 rzl@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-syntaxhighlight: apply
  • 17:29 rzl@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-syntaxhighlight: apply
  • 17:23 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1080-1081,1084-1087,1093-1095,1113-1118].eqiad.wmnet
  • 17:22 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1065-1079].eqiad.wmnet
  • 17:22 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1065-1079].eqiad.wmnet
  • 17:13 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1065-1079].eqiad.wmnet
  • 17:13 rzl@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 17:11 rzl@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 17:10 rzl@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 17:08 rzl@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 17:05 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1065-1079].eqiad.wmnet
  • 17:04 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1044-1057,1064].eqiad.wmnet
  • 17:04 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1044-1057,1064].eqiad.wmnet
  • 16:56 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1044-1057,1064].eqiad.wmnet
  • 16:54 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
  • 16:54 kamila@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts deploy2002.codfw.wmnet
  • 16:54 kamila@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 16:54 kamila@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: deploy2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - kamila@cumin1004"
  • 16:54 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply
  • 16:53 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=codfw,name=dse-k8s-wdqs2005.codfw.wmnet
  • 16:53 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=codfw,name=dse-k8s-wdqs2005.codfw.wmnet
  • 16:53 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-wdqs1004.eqiad.wmnet
  • 16:53 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-wdqs1004.eqiad.wmnet
  • 16:52 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-sre: apply
  • 16:52 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-sre: apply
  • 16:51 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-sre: apply
  • 16:51 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-sre: apply
  • 16:50 kamila@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: deploy2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - kamila@cumin1004"
  • 16:50 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-sre: apply
  • 16:50 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-sre: apply
  • 16:46 kamila@cumin1004: START - Cookbook sre.dns.netbox
  • 16:43 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1044-1057,1064].eqiad.wmnet
  • 16:42 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1006-1007,1015-1016,1021,1034-1043].eqiad.wmnet
  • 16:42 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1006-1007,1015-1016,1021,1034-1043].eqiad.wmnet
  • 16:42 kamila@dns7001: END - running authdns-update
  • 16:41 kamila@cumin1004: START - Cookbook sre.hosts.decommission for hosts deploy2002.codfw.wmnet
  • 16:39 kamila@dns7001: START - running authdns-update
  • 16:37 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm-batch' for release 'main' .
  • 16:33 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1006-1007,1015-1016,1021,1034-1043].eqiad.wmnet
  • 16:26 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-wdqs1004.eqiad.wmnet
  • 16:24 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1006-1007,1015-1016,1021,1034-1043].eqiad.wmnet
  • 16:24 jasmine@cumin2003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-worker-eqiad
  • 16:21 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm-batch' for release 'main' .
  • 16:20 vgutierrez@cumin2003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P{lvs5004.*} and A:liberica
  • 16:19 vgutierrez@cumin2003: START - Cookbook sre.loadbalancer.admin config_reloading P{lvs5004.*} and A:liberica
  • 16:19 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-wdqs1004.eqiad.wmnet
  • 16:17 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-wdqs2005.codfw.wmnet
  • 16:11 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-wdqs2005.codfw.wmnet
  • 16:07 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'.
  • 16:07 moritzm: installing apache2 security updates
  • 16:06 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'.
  • 15:56 vgutierrez@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host lvs5004.eqsin.wmnet with OS trixie
  • 15:54 amastilovic@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/blunderbuss: apply
  • 15:52 amastilovic@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/blunderbuss: apply
  • 15:41 slyngshede@deploy1003: Finished scap sync-world: Backport for debug.json: order codfw (primary) DC backends first (T439674) (duration: 11m 20s)
  • 15:35 slyngshede@deploy1003: slyngshede: Continuing with deployment
  • 15:35 vgutierrez@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs5004.eqsin.wmnet with reason: host reimage
  • 15:34 slyngshede@deploy1003: slyngshede: Backport for debug.json: order codfw (primary) DC backends first (T439674) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 15:31 vgutierrez@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on lvs5004.eqsin.wmnet with reason: host reimage
  • 15:31 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 15:31 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 15:30 slyngshede@deploy1003: Started scap sync-world: Backport for debug.json: order codfw (primary) DC backends first (T439674)
  • 15:29 slyngshede@cumin1004: END (PASS) - Cookbook sre.switchdc.mediawiki.09-unlock-scap (exit_code=0) for datacenter switchover from eqiad to codfw
  • 15:29 root@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: Datacenter switchover from eqiad to codfw - (duration: 126m 08s)
  • 15:28 root@deploy1003: Forcefully removed global lock: Datacenter switchover from eqiad to codfw -
  • 15:28 slyngshede@cumin1004: START - Cookbook sre.switchdc.mediawiki.09-unlock-scap for datacenter switchover from eqiad to codfw
  • 15:28 slyngshede@cumin1004: END (PASS) - Cookbook sre.switchdc.mediawiki.09-run-puppet-on-db-masters (exit_code=0) for datacenter switchover from eqiad to codfw
  • 15:27 vgutierrez@cumin2003: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs5004.eqsin.wmnet (remove config only)
  • 15:27 vgutierrez@cumin2003: START - Cookbook sre.hosts.dhcp for host lvs5004.eqsin.wmnet (remove config only)
  • 15:23 tchin@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventstreams: apply
  • 15:22 tchin@deploy1003: helmfile [codfw] START helmfile.d/services/eventstreams: apply
  • 15:22 tchin@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventstreams: apply
  • 15:22 ebysans@deploy1003: helmfile [codfw] DONE helmfile.d/services/edit-analytics: apply
  • 15:22 ebysans@deploy1003: helmfile [codfw] START helmfile.d/services/edit-analytics: apply
  • 15:21 ebysans@deploy1003: helmfile [eqiad] DONE helmfile.d/services/edit-analytics: apply
  • 15:21 tchin@deploy1003: helmfile [codfw] START helmfile.d/services/eventstreams: apply
  • 15:21 ebysans@deploy1003: helmfile [eqiad] START helmfile.d/services/edit-analytics: apply
  • 15:21 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
  • 15:20 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
  • 15:19 slyngshede@dns1004: END - running authdns-update
  • 15:17 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
  • 15:16 slyngshede@dns1004: START - running authdns-update
  • 15:16 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply
  • 15:16 slyngshede@cumin1004: START - Cookbook sre.switchdc.mediawiki.09-run-puppet-on-db-masters for datacenter switchover from eqiad to codfw
  • 15:13 slyngshede@dns1004: END - running authdns-update
  • 15:11 slyngshede@dns1004: START - running authdns-update
  • 15:10 slyngshede@cumin1004: END (PASS) - Cookbook sre.switchdc.mediawiki.09-restore-ttl (exit_code=0) for datacenter switchover from eqiad to codfw
  • 15:10 slyngshede@cumin1004: START - Cookbook sre.switchdc.mediawiki.09-restore-ttl for datacenter switchover from eqiad to codfw
  • 15:10 slyngshede@cumin1004: END (PASS) - Cookbook sre.switchdc.mediawiki.08-start-maintenance (exit_code=0) for datacenter switchover from eqiad to codfw
  • 15:09 root@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-cron: apply
  • 15:08 root@deploy1003: helmfile [codfw] START helmfile.d/services/mw-cron: apply
  • 15:07 root@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-cron: apply
  • 15:06 root@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-cron: apply
  • 15:06 slyngshede@cumin1004: START - Cookbook sre.switchdc.mediawiki.08-start-maintenance for datacenter switchover from eqiad to codfw
  • 15:06 slyngshede@cumin1004: END (PASS) - Cookbook sre.switchdc.mediawiki.08-restart-mw-jobrunner (exit_code=0) for datacenter switchover from eqiad to codfw
  • 15:06 root@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-jobrunner: sync
  • 15:06 root@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-jobrunner: sync
  • 15:06 slyngshede@cumin1004: START - Cookbook sre.switchdc.mediawiki.08-restart-mw-jobrunner for datacenter switchover from eqiad to codfw
  • 15:05 slyngshede@cumin1004: END (PASS) - Cookbook sre.switchdc.mediawiki.07-set-readwrite (exit_code=0) for datacenter switchover from eqiad to codfw
  • 15:05 slyngshede@cumin1004: MediaWiki read-only period ends at: 2026-10-28 15:05:31.302917
  • 15:02 slyngshede@cumin1004: START - Cookbook sre.switchdc.mediawiki.02-set-readonly for datacenter switchover from eqiad to codfw
  • 14:56 slyngshede@cumin1004: END (PASS) - Cookbook sre.switchdc.mediawiki.01-stop-maintenance (exit_code=0) for datacenter switchover from eqiad to codfw
  • 14:54 slyngshede@cumin1004: START - Cookbook sre.switchdc.mediawiki.01-stop-maintenance for datacenter switchover from eqiad to codfw
  • 14:52 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm-batch' for release 'main' .
  • 14:49 slyngshede@cumin1004: END (PASS) - Cookbook sre.switchdc.mediawiki.00-downtime-db-readonly-checks (exit_code=0) for datacenter switchover from eqiad to codfw
  • 14:49 slyngshede@cumin1004: START - Cookbook sre.switchdc.mediawiki.00-downtime-db-readonly-checks for datacenter switchover from eqiad to codfw
  • 14:49 slyngshede@cumin1004: END (PASS) - Cookbook sre.switchdc.mediawiki.00-reduce-ttl (exit_code=0) for datacenter switchover from eqiad to codfw
  • 14:43 slyngshede@cumin1004: START - Cookbook sre.switchdc.mediawiki.00-reduce-ttl for datacenter switchover from eqiad to codfw
  • 14:43 slyngshede@cumin1004: END (ERROR) - Cookbook sre.switchdc.mediawiki.00-optional-warmup-caches (exit_code=97) for datacenter switchover from eqiad to codfw
  • 14:42 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm-batch' for release 'main' .
  • 14:41 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: apply
  • 14:41 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: apply
  • 14:40 vgutierrez@cumin2003: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs5004.eqsin.wmnet (push config only)
  • 14:40 vgutierrez@cumin2003: START - Cookbook sre.hosts.dhcp for host lvs5004.eqsin.wmnet (push config only)
  • 14:40 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on asw2-b-eqiad,cr[1-2]-eqiad,ssw1-a[1,8]-eqiad with reason: extending so down port alerts dont fire
  • 14:39 vgutierrez@cumin2003: START - Cookbook sre.hosts.reimage for host lvs5004.eqsin.wmnet with OS trixie
  • 14:38 cmooney@cumin1004: DONE (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 3:00:00 on asw2-b2-eqiad,cr[1-2]-eqiad,ssw1-a[1,8]-eqiad with reason: extending so down port alerts dont fire
  • 14:38 vgutierrez@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host lvs5004.eqsin.wmnet with OS trixie
  • 14:38 vgutierrez@cumin2003: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs5004.eqsin.wmnet (remove config only)
  • 14:38 vgutierrez@cumin2003: START - Cookbook sre.hosts.dhcp for host lvs5004.eqsin.wmnet (remove config only)
  • 14:37 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm-batch' for release 'main' .
  • 14:33 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm-batch' for release 'main' .
  • 14:31 tchin@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventstreams: apply
  • 14:30 tchin@deploy1003: helmfile [codfw] START helmfile.d/services/eventstreams: apply
  • 14:30 ebysans@deploy1003: helmfile [staging] DONE helmfile.d/services/edit-analytics: apply
  • 14:30 ebysans@deploy1003: helmfile [staging] START helmfile.d/services/edit-analytics: apply
  • 14:29 tchin@deploy1003: helmfile [staging] DONE helmfile.d/services/eventstreams: apply
  • 14:29 tchin@deploy1003: helmfile [staging] START helmfile.d/services/eventstreams: apply
  • 14:27 slyngshede@cumin1004: START - Cookbook sre.switchdc.mediawiki.00-optional-warmup-caches for datacenter switchover from eqiad to codfw
  • 14:22 vgutierrez@cumin2003: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs5004.eqsin.wmnet (push config only)
  • 14:22 vgutierrez@cumin2003: START - Cookbook sre.hosts.dhcp for host lvs5004.eqsin.wmnet (push config only)
  • 14:21 vgutierrez@cumin2003: START - Cookbook sre.hosts.reimage for host lvs5004.eqsin.wmnet with OS trixie
  • 14:18 btullis@dns1004: END - running authdns-update
  • 14:15 btullis@dns1004: START - running authdns-update
  • 14:15 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 14:15 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update subint entries for row A on cr2-eqiad - cmooney@cumin1004"
  • 14:15 jgreen@dns2004: START - running authdns-update
  • 14:14 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update subint entries for row A on cr2-eqiad - cmooney@cumin1004"
  • 14:12 jgreen@dns2004: START - running authdns-update
  • 14:11 btullis@dns1004: START - running authdns-update
  • 14:09 cmooney@cumin1004: START - Cookbook sre.dns.netbox
  • 14:06 jforrester@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mathoid: apply
  • 14:06 bjensen: homer "lsw1-e3-codfw" commit "remove wikikube-worker2337 (T442714)"
  • 14:05 jforrester@deploy1003: helmfile [eqiad] START helmfile.d/services/mathoid: apply
  • 14:05 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/mathoid: apply
  • 14:05 tchin@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/eventstreams-internal: apply
  • 14:04 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/mathoid: apply
  • 14:04 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/mathoid: apply
  • 14:04 tchin@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/eventstreams-internal: apply
  • 14:04 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/mathoid: apply
  • 14:00 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm-batch' for release 'main' .
  • 13:29 klausman@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'.
  • 13:28 klausman@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'.
  • 13:24 topranks: move asw2-a-eqiad ae2 ports into ae1 and enable the far-end on ssw1-a8-eqiad where the fibres have been moved T441014
  • 13:22 slyngshede@cumin1004: END (PASS) - Cookbook sre.switchdc.mediawiki.00-lock-scap (exit_code=0) for datacenter switchover from eqiad to codfw
  • 13:22 root@deploy1003: Locking from deployment [ALL REPOSITORIES]: Datacenter switchover from eqiad to codfw -
  • 13:22 slyngshede@cumin1004: START - Cookbook sre.switchdc.mediawiki.00-lock-scap for datacenter switchover from eqiad to codfw
  • 13:22 klausman@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'.
  • 13:21 cwilliams@cumin1004: END (PASS) - Cookbook sre.backup.offsite (exit_code=0)
  • 13:21 cwilliams@cumin1004: START - Cookbook sre.backup.offsite
  • 13:21 cwilliams@cumin1004: END (PASS) - Cookbook sre.backup.offsite (exit_code=0)
  • 13:21 cwilliams@cumin1004: START - Cookbook sre.backup.offsite
  • 13:20 klausman@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'.
  • 13:19 Dreamy_Jazz: Afternoon UTC backport window done
  • 13:18 mlitn@deploy1003: Finished scap sync-world: Backport for Squashed diff to master (minus i18n) (1.47.0-wmf.24, 1.48.0-wmf.1) (duration: 11m 05s)
  • 13:16 cwilliams@cumin1004: END (PASS) - Cookbook sre.backup.offsite (exit_code=0)
  • 13:16 cwilliams@cumin1004: START - Cookbook sre.backup.offsite
  • 13:15 blake@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host wikikube-worker2337.codfw.wmnet
  • 13:12 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2337.codfw.wmnet
  • 13:12 mlitn@deploy1003: mlitn: Continuing with deployment
  • 13:11 mlitn@deploy1003: mlitn: Backport for Squashed diff to master (minus i18n) (1.47.0-wmf.24, 1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 13:09 Dreamy_Jazz: Running `foreachwikiindblist group0.dblist cleanupWatchlistTemporaryAccountRows.php` for T439059
  • 13:09 Dreamy_Jazz: Running `foreachwikiindblist group0.dblist cleanupWatchlistTemporaryAccountRows.php --wiki=testwiki` for T439059
  • 13:08 klausman@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'.
  • 13:07 klausman@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'.
  • 13:06 mlitn@deploy1003: Started scap sync-world: Backport for Squashed diff to master (minus i18n) (1.47.0-wmf.24, 1.48.0-wmf.1)
  • 13:02 dreamyjazz@deploy1003: Finished scap sync-world: Backport for Add update maintenance script to drop watchlist rows for temp users (T439059) (1.48.0-wmf.1) (duration: 13m 59s)
  • 12:58 kart_: Updated cxserver to 2026-10-28-122929-production
  • 12:57 kartik@deploy1003: helmfile [eqiad] DONE helmfile.d/services/cxserver: apply
  • 12:57 kartik@deploy1003: helmfile [eqiad] START helmfile.d/services/cxserver: apply
  • 12:56 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment
  • 12:54 kartik@deploy1003: helmfile [codfw] DONE helmfile.d/services/cxserver: apply
  • 12:53 kartik@deploy1003: helmfile [codfw] START helmfile.d/services/cxserver: apply
  • 12:52 dreamyjazz@deploy1003: dreamyjazz: Backport for Add update maintenance script to drop watchlist rows for temp users (T439059) (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 12:51 kartik@deploy1003: helmfile [staging] DONE helmfile.d/services/cxserver: apply
  • 12:51 topranks: disable ae1 on cr2-eqiad connecting to asw2-a-eqiad (traffic is flowing via cr1-eqiad) T441014
  • 12:51 kartik@deploy1003: helmfile [staging] START helmfile.d/services/cxserver: apply
  • 12:48 dreamyjazz@deploy1003: Started scap sync-world: Backport for Add update maintenance script to drop watchlist rows for temp users (T439059) (1.48.0-wmf.1)
  • 12:40 hnowlan@cumin1004: END (PASS) - Cookbook sre.misc-clusters.roll-restart-restbase (exit_code=0) rolling restart_daemons on A:restbase-eqiad
  • 12:36 hnowlan@cumin1004: END (PASS) - Cookbook sre.misc-clusters.roll-restart-restbase (exit_code=0) rolling restart_daemons on A:restbase-codfw
  • 12:34 topranks: make cr1-eqiad take over VRRP for the remaining row A vlans
  • 12:32 hnowlan@cumin1004: START - Cookbook sre.misc-clusters.roll-restart-restbase rolling restart_daemons on A:restbase-eqiad
  • 12:32 topranks: revert VRRP changes on cr2-eqiad so cr1-eqiad takes over for some of eqiad row A subnets T441014
  • 12:31 mvolz@deploy1003: Finished scap deploy-service citoid: update (duration: 50m 46s)
  • 12:29 mvolz@deploy1003: scap deploy-service citoid: [PRODUCTION/DEFAULT] eqiad Deploying citoid
  • 12:28 hnowlan@cumin1004: START - Cookbook sre.misc-clusters.roll-restart-restbase rolling restart_daemons on A:restbase-codfw
  • 12:26 vgutierrez@cumin2003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P{lvs5005.*} and A:liberica
  • 12:26 vgutierrez@cumin2003: START - Cookbook sre.loadbalancer.admin config_reloading P{lvs5005.*} and A:liberica
  • 12:25 topranks: re-enabling et-0/1/0 sub-interfaces for row A vlans on cr1-eqiad (they are now trunked on asw2-a-eqiad side to ssw1-a1-eqiad so no repeat of last time expected) T441014
  • 12:23 vgutierrez@cumin2003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P{lvs5004.*} and A:liberica
  • 12:23 vgutierrez@cumin2003: START - Cookbook sre.loadbalancer.admin config_reloading P{lvs5004.*} and A:liberica
  • 12:22 hnowlan@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) statsd.eqiad.wmnet on all recursors
  • 12:22 hnowlan@cumin1004: START - Cookbook sre.dns.wipe-cache statsd.eqiad.wmnet on all recursors
  • 12:21 topranks: trunk all row A vlans from asw2-a-eqiad ae1 to ssw1-a1-eqiad lag1 T441014
  • 12:20 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 12:20 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update reverse dns for moved IPs on cr1-eqiad - cmooney@cumin1004"
  • 12:20 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update reverse dns for moved IPs on cr1-eqiad - cmooney@cumin1004"
  • 12:15 cmooney@cumin1004: START - Cookbook sre.dns.netbox
  • 12:12 hnowlan@cumin1004: END (PASS) - Cookbook sre.misc-clusters.roll-restart-restbase (exit_code=0) rolling restart_daemons on A:restbase
  • 12:01 vgutierrez@cumin2003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P{lvs5005.*} and A:liberica
  • 12:00 vgutierrez@cumin2003: START - Cookbook sre.loadbalancer.admin config_reloading P{lvs5005.*} and A:liberica
  • 12:00 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
  • 12:00 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
  • 11:56 hnowlan@cumin1004: START - Cookbook sre.misc-clusters.roll-restart-restbase rolling restart_daemons on A:restbase
  • 11:52 Amir1: mwscript-k8s --follow --dblist=ores -- extensions/ORES/maintenance/PurgeScoreCache.php --model damaging --old (T440358)
  • 11:46 vgutierrez@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host lvs5005.eqsin.wmnet with OS trixie
  • 11:46 mvolz@deploy1003: scap deploy-service citoid: [PRODUCTION/DEFAULT] codfw Deploying citoid
  • 11:42 hnowlan@dns1004: END - running authdns-update
  • 11:40 mvolz@deploy1003: scap deploy-service citoid: [STAGING/DEFAULT] staging-eqiad Deploying citoid
  • 11:40 mvolz@deploy1003: Started scap deploy-service citoid: update
  • 11:39 hnowlan@dns1004: START - running authdns-update
  • 11:35 cwilliams@cumin1004: END (PASS) - Cookbook sre.backup.offsite (exit_code=0)
  • 11:34 cwilliams@cumin1004: START - Cookbook sre.backup.offsite
  • 11:29 dbrant@deploy1003: helmfile [codfw] DONE helmfile.d/services/mobileapps: apply
  • 11:28 dbrant@deploy1003: helmfile [codfw] START helmfile.d/services/mobileapps: apply
  • 11:28 dbrant@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply
  • 11:28 hnowlan@dns1004: END - running authdns-update
  • 11:27 dbrant@deploy1003: helmfile [eqiad] START helmfile.d/services/mobileapps: apply
  • 11:27 dbrant@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
  • 11:27 dbrant@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
  • 11:25 hnowlan@dns1004: START - running authdns-update
  • 11:23 vgutierrez@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs5005.eqsin.wmnet with reason: host reimage
  • 11:23 cwilliams@cumin1004: END (PASS) - Cookbook sre.backup.offsite (exit_code=0)
  • 11:22 cwilliams@cumin1004: START - Cookbook sre.backup.offsite
  • 11:19 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wikidata: apply
  • 11:18 vgutierrez@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on lvs5005.eqsin.wmnet with reason: host reimage
  • 11:18 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wikidata: apply
  • 11:14 vgutierrez@cumin2003: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs5005.eqsin.wmnet (remove config only)
  • 11:14 vgutierrez@cumin2003: START - Cookbook sre.hosts.dhcp for host lvs5005.eqsin.wmnet (remove config only)
  • 11:12 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user resquito
  • 11:12 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user resquito
  • 11:08 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user joelyrookewmde
  • 11:08 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user joelyrookewmde
  • 11:07 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user joelyrookewmde
  • 11:06 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user joelyrookewmde
  • 11:03 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user arinaigum
  • 11:02 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user arinaigum
  • 10:57 topranks: shut down cr1-eqiad links connecting asw2-a-eqiad to shift traffic to cr2-eqiad equivalent T441014
  • 10:55 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on 6 hosts with reason: migrate row a/b network uplinkks from CRs to new Nokia spines
  • 10:50 vgutierrez@cumin2003: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs5005.eqsin.wmnet (push config only)
  • 10:50 vgutierrez@cumin2003: START - Cookbook sre.hosts.dhcp for host lvs5005.eqsin.wmnet (push config only)
  • 10:49 vgutierrez@cumin2003: START - Cookbook sre.hosts.reimage for host lvs5005.eqsin.wmnet with OS trixie
  • 10:46 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user west1
  • 10:46 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user west1
  • 10:46 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1046.eqiad.wmnet
  • 10:46 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1046.eqiad.wmnet
  • 10:42 vgutierrez@cumin2003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P{lvs5005.*} and A:liberica
  • 10:42 vgutierrez@cumin2003: START - Cookbook sre.loadbalancer.admin config_reloading P{lvs5005.*} and A:liberica
  • 10:40 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti1046.eqiad.wmnet
  • 10:36 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1046.eqiad.wmnet
  • 10:34 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/blunderbuss: apply
  • 10:33 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/blunderbuss: apply
  • 10:32 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host apt2002.wikimedia.org
  • 10:30 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2047.codfw.wmnet
  • 10:30 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2047.codfw.wmnet
  • 10:26 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host apt2002.wikimedia.org
  • 10:24 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti2047.codfw.wmnet
  • 10:24 moritzm: installing Perl security updates
  • 10:21 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2047.codfw.wmnet
  • 10:09 dpogorzelski@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply
  • 10:08 dpogorzelski@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply
  • 10:08 dpogorzelski@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply
  • 10:08 dpogorzelski@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply
  • 10:07 dpogorzelski@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply
  • 10:07 dpogorzelski@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply
  • 09:55 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user amire80
  • 09:55 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user amire80
  • 09:52 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user amire80
  • 09:52 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user amire80
  • 09:51 javiermonton@deploy1003: Finished scap deploy-service pageview-trending-relative-next: Test Flink checks in spiderpig (duration: 01m 21s)
  • 09:51 ladsgroup@deploy1003: Finished scap sync-world: Backport for Fix two bugs on PurgeScoreCache (T440358) (1.47.0-wmf.24, 1.48.0-wmf.1) (duration: 11m 20s)
  • 09:50 javiermonton@deploy1003: scap deploy-service pageview-trending-relative-next: [PRODUCTION/DEFAULT] dse-k8s-eqiad Deploying pageview-trending-relative-next
  • 09:50 javiermonton@deploy1003: Started scap deploy-service pageview-trending-relative-next: Test Flink checks in spiderpig
  • 09:50 moritzm: installing nodejs security updates on bookworm
  • 09:50 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply
  • 09:49 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply
  • 09:48 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply
  • 09:48 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply
  • 09:47 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply
  • 09:47 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply
  • 09:45 ladsgroup@deploy1003: ladsgroup: Continuing with deployment
  • 09:44 ladsgroup@deploy1003: ladsgroup: Backport for Fix two bugs on PurgeScoreCache (T440358) (1.47.0-wmf.24, 1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 09:39 ladsgroup@deploy1003: Started scap sync-world: Backport for Fix two bugs on PurgeScoreCache (T440358) (1.47.0-wmf.24, 1.48.0-wmf.1)
  • 09:30 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2033.codfw.wmnet
  • 09:30 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2033.codfw.wmnet
  • 09:25 kevinbazira@deploy1003: helmfile [ml-serve-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' .
  • 09:24 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'tts-section-generator' for release 'main' .
  • 09:23 moritzm: installing jq security updates
  • 09:22 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply
  • 09:22 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply
  • 09:22 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti2033.codfw.wmnet
  • 09:22 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply
  • 09:22 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply
  • 09:22 kevinbazira@deploy1003: helmfile [ml-staging-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' .
  • 09:22 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply
  • 09:22 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply
  • 09:14 javiermonton@deploy1003: Finished scap deploy-service pageview-trending-relative-next: testing checks for Flink deployments. (duration: 03m 58s)
  • 09:12 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2033.codfw.wmnet
  • 09:10 javiermonton@deploy1003: scap deploy-service pageview-trending-relative-next: [PRODUCTION/DEFAULT] dse-k8s-eqiad Deploying pageview-trending-relative-next
  • 09:10 javiermonton@deploy1003: Started scap deploy-service pageview-trending-relative-next: testing checks for Flink deployments.
  • 08:58 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user niharika29
  • 08:58 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user niharika29
  • 08:49 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user niharika29
  • 08:49 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user niharika29
  • 08:48 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user niharika29
  • 08:48 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user niharika29
  • 08:48 moritzm: failover Ganeti master in codfw02 to ganeti2034
  • 08:46 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revision-models' for release 'main' .
  • 08:45 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2034.codfw.wmnet
  • 08:45 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2034.codfw.wmnet
  • 08:44 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revision-models' for release 'main' .
  • 08:40 vgutierrez: repooling cp2049
  • 08:36 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti2034.codfw.wmnet
  • 08:36 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2034.codfw.wmnet
  • 08:35 jmm@cumin2003: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti2034.codfw.wmnet
  • 08:35 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2034.codfw.wmnet
  • 08:33 moritzm: installing perl security updates
  • 08:17 moritzm: installing clamav security updates
  • 08:17 jmm@cumin2003: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti2034.codfw.wmnet
  • 08:16 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2034.codfw.wmnet
  • 08:12 moritzm: failover Ganeti master in codfw to ganeti2048
  • 08:09 moritzm: failover Ganeti master in eqiad to ganeti1048
  • 07:36 hashar: Restarted Gerrit on gerrit1003
  • 07:22 hashar@deploy1003: Finished deploy [gerrit/gerrit@b02d300]: Update Zuul plugin to align CSS styling - T441684 (duration: 00m 14s)
  • 07:22 hashar@deploy1003: Started deploy [gerrit/gerrit@b02d300]: Update Zuul plugin to align CSS styling - T441684
  • 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 37s)
  • 02:01 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image
  • 01:33 kemayo@deploy1003: helmfile [eqiad] DONE helmfile.d/services/editcheck-headless: apply
  • 01:32 kemayo@deploy1003: helmfile [eqiad] START helmfile.d/services/editcheck-headless: apply
  • 01:27 kemayo@deploy1003: helmfile [codfw] DONE helmfile.d/services/editcheck-headless: apply
  • 01:25 kemayo@deploy1003: helmfile [codfw] START helmfile.d/services/editcheck-headless: apply
  • 01:21 kemayo@deploy1003: helmfile [staging] DONE helmfile.d/services/editcheck-headless: apply
  • 01:20 kemayo@deploy1003: helmfile [staging] START helmfile.d/services/editcheck-headless: apply
  • 01:19 kemayo@deploy1003: helmfile [staging] DONE helmfile.d/services/editcheck-headless: apply
  • 01:19 kemayo@deploy1003: helmfile [staging] START helmfile.d/services/editcheck-headless: apply
  • 01:08 rzl@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 01:07 rzl@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 01:01 kemayo@deploy1003: helmfile [staging] DONE helmfile.d/services/editcheck-headless: apply
  • 01:01 kemayo@deploy1003: helmfile [staging] START helmfile.d/services/editcheck-headless: apply
  • 00:20 rzl@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 00:20 rzl@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 00:00 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
  • 00:00 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
  • 00:00 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
  • 00:00 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: sync

2026-10-27

  • 23:49 ladsgroup@deploy1003: Finished scap sync-world: Backport for ffmpeg: Actually apply the -speed argument (1.48.0-wmf.1, 1.47.0-wmf.24) (duration: 11m 08s)
  • 23:43 ladsgroup@deploy1003: ladsgroup: Continuing with deployment
  • 23:42 ladsgroup@deploy1003: ladsgroup: Backport for ffmpeg: Actually apply the -speed argument (1.48.0-wmf.1, 1.47.0-wmf.24) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 23:41 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen: apply
  • 23:40 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen: apply
  • 23:38 ladsgroup@deploy1003: Started scap sync-world: Backport for ffmpeg: Actually apply the -speed argument (1.48.0-wmf.1, 1.47.0-wmf.24)
  • 23:23 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply
  • 23:23 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply
  • 22:49 mutante: planet2003,planet1003 - re-enabled puppet after gerrit:1359406 T442071 and being disabled for somet ime
  • 22:07 jdlrobson@deploy1003: Finished scap sync-world: Backport for Fix typo in config name (T441705) (duration: 14m 28s)
  • 21:59 jdlrobson@deploy1003: jdlrobson: Continuing with deployment
  • 21:58 jdlrobson@deploy1003: jdlrobson: Backport for Fix typo in config name (T441705) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 21:53 jdlrobson@deploy1003: Started scap sync-world: Backport for Fix typo in config name (T441705)
  • 21:51 jdlrobson@deploy1003: Finished scap sync-world: Backport for Enable Vector 2022 CTA on test wikis (T441705) (duration: 35m 00s)
  • 21:40 jdlrobson@deploy1003: jdlrobson: Continuing with deployment
  • 21:36 jdlrobson@deploy1003: jdlrobson: Backport for Enable Vector 2022 CTA on test wikis (T441705) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 21:16 jdlrobson@deploy1003: Started scap sync-world: Backport for Enable Vector 2022 CTA on test wikis (T441705)
  • 21:14 andrewbogott: sudo cumin "gitlab-runner*" "systemctl restart ferm" on Tyler's request
  • 21:09 jdlrobson@deploy1003: Finished scap sync-world: Backport for Merge master into wmf.1 (T438390 T438405 T438411 T438413 T440609 T440858 T440995 T441249 T441468 T441477 T441696 T441698 T441700 T441702 T441797 T441918 T442075 T442419 T442585) (1.48.0-wmf.1) (duration: 05m 37s)
  • 21:07 kamila@cumin1004: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=99) for hosts deploy2002.codfw.wmnet
  • 21:07 jdlrobson@deploy1003: jdlrobson: Continuing with deployment
  • 21:06 kamila@cumin1004: START - Cookbook sre.hosts.decommission for hosts deploy2002.codfw.wmnet
  • 21:05 jdlrobson@deploy1003: jdlrobson: Backport for Merge master into wmf.1 (T438390 T438405 T438411 T438413 T440609 T440858 T440995 T441249 T441468 T441477 T441696 T441698 T441700 T441702 T441797 T441918 T442075 T442419 T442585) (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 21:03 jdlrobson@deploy1003: Started scap sync-world: Backport for Merge master into wmf.1 (T438390 T438405 T438411 T438413 T440609 T440858 T440995 T441249 T441468 T441477 T441696 T441698 T441700 T441702 T441797 T441918 T442075 T442419 T442585) (1.48.0-wmf.1)
  • 20:41 brett@cumin2003: END (PASS) - Cookbook sre.cdn.roll-upgrade-ats (exit_code=0) Rolling upgrade of ATS on A:cp-text_codfw and A:cp - 9.2.16 upgrade (T441945)
  • 20:39 bking@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 20:39 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: fix IPv6 PTR - bking@cumin2003"
  • 20:39 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: fix IPv6 PTR - bking@cumin2003"
  • 20:32 bking@cumin2003: START - Cookbook sre.dns.netbox
  • 20:31 ayounsi@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 20:31 ayounsi@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: normalize - ayounsi@cumin1004"
  • 20:28 hashar: arwiki: mwscript emptyUserGroup.php --wiki=arwiki uploader --create-log --log-reason 'uploader user group merged into autoreview - T442578'
  • 20:27 ayounsi@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: normalize - ayounsi@cumin1004"
  • 20:25 hashar@deploy1003: Finished scap sync-world: Backport for [arwiki] Remove uploader user group and merge it with autoreview (T442578) (duration: 14m 07s)
  • 20:22 ayounsi@cumin1004: START - Cookbook sre.dns.netbox
  • 20:20 robh@cumin1004: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts sretest2005.codfw.wmnet
  • 20:20 robh@cumin1004: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
  • 20:19 robh@cumin1004: END (FAIL) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=99) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: sretest2005.codfw.wmnet decommissioned, removing all IPs except the asset tag one - robh@cumin1004"
  • 20:19 hashar@deploy1003: gergesshamon, hashar: Continuing with deployment
  • 20:17 robh@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: sretest2005.codfw.wmnet decommissioned, removing all IPs except the asset tag one - robh@cumin1004"
  • 20:15 hashar@deploy1003: gergesshamon, hashar: Backport for [arwiki] Remove uploader user group and merge it with autoreview (T442578) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 20:14 rzl: rzl@apt1002:~$ sudo -i reprepro copy bookworm-wikimedia bullseye-wikimedia kubeconform
  • 20:13 brett@cumin2003: END (PASS) - Cookbook sre.cdn.roll-upgrade-ats (exit_code=0) Rolling upgrade of ATS on A:cp-upload_codfw and A:cp - 9.2.16 upgrade (T441945)
  • 20:11 hashar@deploy1003: Started scap sync-world: Backport for [arwiki] Remove uploader user group and merge it with autoreview (T442578)
  • 20:11 robh@cumin1004: START - Cookbook sre.dns.netbox
  • 20:07 robh@cumin1004: START - Cookbook sre.hosts.decommission for hosts sretest2005.codfw.wmnet
  • 19:48 sukhe@cumin1004: END (PASS) - Cookbook sre.dns.roll-restart-reboot-wikimedia-dns (exit_code=0) rolling restart_daemons on A:wikidough
  • 19:46 brett@cumin2003: START - Cookbook sre.cdn.roll-upgrade-ats Rolling upgrade of ATS on A:cp-upload_codfw and A:cp - 9.2.16 upgrade (T441945)
  • 19:46 brett@cumin2003: START - Cookbook sre.cdn.roll-upgrade-ats Rolling upgrade of ATS on A:cp-text_codfw and A:cp - 9.2.16 upgrade (T441945)
  • 19:35 sukhe@cumin1004: START - Cookbook sre.dns.roll-restart-reboot-wikimedia-dns rolling restart_daemons on A:wikidough
  • 19:17 sandraebele: Deployed Refinery using Scap, and then deployed onto hdfs
  • 19:08 ebysans@deploy1003: Finished deploy [analytics/refinery@4a515e4]: Deploying Refinery Patch 1355145 [analytics/refinery@4a515e4f] (duration: 05m 03s)
  • 19:03 ebysans@deploy1003: Started deploy [analytics/refinery@4a515e4]: Deploying Refinery Patch 1355145 [analytics/refinery@4a515e4f]
  • 18:57 ebysans@deploy1003: Finished deploy [analytics/refinery@4a515e4] (hadoop-test): Deploying Refinery Patch 1355145 and Ticket T434136 (duration: 00m 43s)
  • 18:56 ebysans@deploy1003: Started deploy [analytics/refinery@4a515e4] (hadoop-test): Deploying Refinery Patch 1355145 and Ticket T434136
  • 18:49 sandraebele: Deploying Refinery at 4a515e4 for change 1355145/T434136 Add per wiki filtering to edits/v3/per_editor
  • 18:46 brett@cumin2003: END (PASS) - Cookbook sre.cdn.roll-upgrade-ats (exit_code=0) Rolling upgrade of ATS on A:cp-text_eqiad and A:cp - 9.2.16 upgrade (T441945)
  • 18:22 brett@cumin2003: END (PASS) - Cookbook sre.cdn.roll-upgrade-ats (exit_code=0) Rolling upgrade of ATS on A:cp-upload_eqiad and A:cp - 9.2.16 upgrade (T441945)
  • 18:13 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=dns4003.*
  • 18:08 otto@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventgate-main: sync
  • 18:08 otto@deploy1003: helmfile [codfw] START helmfile.d/services/eventgate-main: sync
  • 18:08 otto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/eventgate-main: sync
  • 18:07 otto@deploy1003: helmfile [eqiad] START helmfile.d/services/eventgate-main: sync
  • 18:05 otto@deploy1003: helmfile [staging] DONE helmfile.d/services/eventgate-main: sync
  • 18:04 ottomata: roll restarting eventgate-main to pick up stream config change
  • 18:04 otto@deploy1003: helmfile [staging] START helmfile.d/services/eventgate-main: sync
  • 18:03 otto@deploy1003: Finished scap sync-world: Backport for EventStreamConfig: Declare the mediawiki.user_thank.v1 stream (T441426) (duration: 11m 41s)
  • 18:00 brett@puppetserver1001: conftool action : set/pooled=no; selector: name=dns4003.*
  • 17:57 otto@deploy1003: otto, ebysans: Continuing with deployment
  • 17:56 otto@deploy1003: otto, ebysans: Backport for EventStreamConfig: Declare the mediawiki.user_thank.v1 stream (T441426) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 17:53 brett@cumin2003: START - Cookbook sre.cdn.roll-upgrade-ats Rolling upgrade of ATS on A:cp-upload_eqiad and A:cp - 9.2.16 upgrade (T441945)
  • 17:53 brett@cumin2003: START - Cookbook sre.cdn.roll-upgrade-ats Rolling upgrade of ATS on A:cp-text_eqiad and A:cp - 9.2.16 upgrade (T441945)
  • 17:51 otto@deploy1003: Started scap sync-world: Backport for EventStreamConfig: Declare the mediawiki.user_thank.v1 stream (T441426)
  • 17:44 rzl: rzl@apt1002:~$ sudo -i reprepro -C component/envoy-future include bookworm-wikimedia /home/rzl/envoyproxy_1.40.1-1_amd64.changes
  • 17:39 akhatun@deploy1003: Failed scap sync-world: Backport for Prefetch the LAC revision_feature_counts artifact on group1 wikipedias (T432604 T436695) (duration: 07m 58s)
  • 17:37 akhatun@deploy1003: akhatun, otto: Rolling back deployment
  • 17:36 akhatun@deploy1003: akhatun, otto: Backport for Prefetch the LAC revision_feature_counts artifact on group1 wikipedias (T432604 T436695) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 17:31 akhatun@deploy1003: Started scap sync-world: Backport for Prefetch the LAC revision_feature_counts artifact on group1 wikipedias (T432604 T436695)
  • 17:31 mutante: codesearch: fixed the zombie process leak in hound - containers now run with --init (1358873) which reaps orphaned git processes; codesearch1001/2001 went from 160k+ zombies to 0 Now applying to codesearch10 (Cloud VPS, serves the current public codesearch), search incomplete for ~1h while it reindexes T268199
  • 17:17 vgutierrez@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host lvs5006.eqsin.wmnet with OS trixie
  • 17:11 dzahn@cumin1004: conftool action : set/pooled=false; selector: dnsdisc=releases,name=codfw
  • 17:09 mutante: confctl: depooling releases.wikimedia.org in codfw - it is wrongly marked active/active but in reality it is not. only releases1003 is behind the releases.discovery.wmnet name - adjusting config to reality.
  • 16:53 vgutierrez@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs5006.eqsin.wmnet with reason: host reimage
  • 16:53 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti3008.esams.wmnet
  • 16:53 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti3008.esams.wmnet
  • 16:52 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1052.eqiad.wmnet
  • 16:52 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1052.eqiad.wmnet
  • 16:48 vgutierrez@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on lvs5006.eqsin.wmnet with reason: host reimage
  • 16:47 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti1052.eqiad.wmnet
  • 16:46 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-serve1016.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 16:44 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti3008.esams.wmnet
  • 16:44 vgutierrez@cumin2003: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs5006.eqsin.wmnet (remove config only)
  • 16:43 vgutierrez@cumin2003: START - Cookbook sre.hosts.dhcp for host lvs5006.eqsin.wmnet (remove config only)
  • 16:42 slyngshede@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) depool eventstreams in eqiad: T439675 - DC Switchover
  • 16:41 elukey@cumin1004: START - Cookbook sre.hosts.provision for host ml-serve1016.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 16:40 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ping2004.codfw.wmnet
  • 16:37 slyngshede@cumin1004: START - Cookbook sre.discovery.service-route depool eventstreams in eqiad: T439675 - DC Switchover
  • 16:36 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ping2004.codfw.wmnet
  • 16:34 slyngshede@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool eventstreams in codfw: T439675 - DC Switchover
  • 16:30 fabfur@cumin1004: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on A:cp-text_esams
  • 16:30 fabfur@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp3075.esams.wmnet
  • 16:29 sukhe@dns1004: END - running authdns-update
  • 16:29 slyngshede@cumin1004: START - Cookbook sre.discovery.service-route pool eventstreams in codfw: T439675 - DC Switchover
  • 16:27 sukhe@dns1004: START - running authdns-update
  • 16:26 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: dc=eqsin,cluster=dnsbox
  • 16:25 slyngshede@cumin1004: END (PASS) - Cookbook sre.discovery.datacenter (exit_code=0) depool all services in eqiad: Datacenter services switchover - T439675
  • 16:22 tchin@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/eventstreams-internal: apply
  • 16:13 vgutierrez@cumin2003: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs5006.eqsin.wmnet (push config only)
  • 16:12 vgutierrez@cumin2003: START - Cookbook sre.hosts.dhcp for host lvs5006.eqsin.wmnet (push config only)
  • 16:12 vgutierrez@cumin2003: START - Cookbook sre.hosts.reimage for host lvs5006.eqsin.wmnet with OS trixie
  • 16:12 tchin@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/eventstreams-internal: apply
  • 16:10 vgutierrez@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host lvs5006.eqsin.wmnet with OS trixie
  • 16:05 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: dc=eqsin,cluster=dnsbox
  • 15:49 fabfur@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp3074.esams.wmnet
  • 15:46 slyngshede@cumin1004: START - Cookbook sre.discovery.datacenter depool all services in eqiad: Datacenter services switchover - T439675
  • 15:38 vgutierrez@cumin2003: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs5006.eqsin.wmnet (push config only)
  • 15:37 vgutierrez@cumin2003: START - Cookbook sre.hosts.dhcp for host lvs5006.eqsin.wmnet (push config only)
  • 15:37 slyngshede@cumin1004: END (FAIL) - Cookbook sre.discovery.service-route (exit_code=99) depool mw-web-ro in eqiad: T439675 - DC Switchover
  • 15:37 vgutierrez@cumin2003: START - Cookbook sre.hosts.reimage for host lvs5006.eqsin.wmnet with OS trixie
  • 15:33 vgutierrez@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host lvs5006.eqsin.wmnet with OS trixie
  • 15:33 vgutierrez@cumin2003: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs5006.eqsin.wmnet (remove config only)
  • 15:33 vgutierrez@cumin2003: START - Cookbook sre.hosts.dhcp for host lvs5006.eqsin.wmnet (remove config only)
  • 15:32 slyngshede@cumin1004: START - Cookbook sre.discovery.service-route depool mw-web-ro in eqiad: T439675 - DC Switchover
  • 15:21 cgoubert@cumin2003: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) check mw-api-ext-ro: maintenance
  • 15:21 cgoubert@cumin2003: START - Cookbook sre.discovery.service-route check mw-api-ext-ro: maintenance
  • 15:16 slyngshede@cumin1004: END (FAIL) - Cookbook sre.discovery.service-route (exit_code=99) depool mw-api-ext-ro in eqiad: T439675 - DC Switchover
  • 15:13 vgutierrez@cumin2003: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs5006.eqsin.wmnet (push config only)
  • 15:13 vgutierrez@cumin2003: START - Cookbook sre.hosts.dhcp for host lvs5006.eqsin.wmnet (push config only)
  • 15:13 vgutierrez@cumin2003: START - Cookbook sre.hosts.reimage for host lvs5006.eqsin.wmnet with OS trixie
  • 15:11 slyngshede@cumin1004: START - Cookbook sre.discovery.service-route depool mw-api-ext-ro in eqiad: T439675 - DC Switchover
  • 15:09 slyngshede@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool eqiad [reason: no reason specified, no task ID specified]
  • 15:08 slyngshede@cumin1004: START - Cookbook sre.dns.admin DNS admin: depool eqiad [reason: no reason specified, no task ID specified]
  • 15:08 vgutierrez@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host lvs5006.eqsin.wmnet with OS trixie
  • 15:08 vgutierrez@cumin2003: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs5006.eqsin.wmnet (remove config only)
  • 15:08 vgutierrez@cumin2003: START - Cookbook sre.hosts.dhcp for host lvs5006.eqsin.wmnet (remove config only)
  • 15:07 fabfur@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp3073.esams.wmnet
  • 15:06 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
  • 15:04 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
  • 14:57 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti3008.esams.wmnet
  • 14:57 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1052.eqiad.wmnet
  • 14:55 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
  • 14:51 krinkle@deploy1003: Finished scap sync-world: Backport for xff.log: Fix jobrunner filter and log whether TrustedXFF was used (T288223) (duration: 13m 23s)
  • 14:51 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply
  • 14:49 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1051.eqiad.wmnet
  • 14:49 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1051.eqiad.wmnet
  • 14:47 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
  • 14:47 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply
  • 14:47 hashar: Restarting Gerrit
  • 14:46 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ping1004.eqiad.wmnet
  • 14:45 krinkle@deploy1003: krinkle: Continuing with deployment
  • 14:44 ayounsi@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
  • 14:44 ayounsi@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
  • 14:43 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti1051.eqiad.wmnet
  • 14:43 vgutierrez@cumin2003: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs5006.eqsin.wmnet (push config only)
  • 14:43 vgutierrez@cumin2003: START - Cookbook sre.hosts.dhcp for host lvs5006.eqsin.wmnet (push config only)
  • 14:43 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ping1004.eqiad.wmnet
  • 14:42 vgutierrez@cumin2003: START - Cookbook sre.hosts.reimage for host lvs5006.eqsin.wmnet with OS trixie
  • 14:42 krinkle@deploy1003: krinkle: Backport for xff.log: Fix jobrunner filter and log whether TrustedXFF was used (T288223) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 14:41 ayounsi@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary
  • 14:41 ayounsi@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
  • 14:38 krinkle@deploy1003: Started scap sync-world: Backport for xff.log: Fix jobrunner filter and log whether TrustedXFF was used (T288223)
  • 14:34 moritzm: installing Django security updates
  • 14:33 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1051.eqiad.wmnet
  • 14:31 vgutierrez@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host lvs5006.eqsin.wmnet with OS trixie
  • 14:31 vgutierrez@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs5006.eqsin.wmnet (remove config only)
  • 14:31 vgutierrez@cumin1004: START - Cookbook sre.hosts.dhcp for host lvs5006.eqsin.wmnet (remove config only)
  • 14:28 fabfur@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp3072.esams.wmnet
  • 14:27 dreamyjazz@deploy1003: Finished scap sync-world: Backport for AbuseReview: Enable queues on several wikis (T442476 T442204) (duration: 17m 13s)
  • 14:20 vgutierrez@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs5006.eqsin.wmnet (push config only)
  • 14:20 vgutierrez@cumin1004: START - Cookbook sre.hosts.dhcp for host lvs5006.eqsin.wmnet (push config only)
  • 14:19 vgutierrez@cumin1004: START - Cookbook sre.hosts.reimage for host lvs5006.eqsin.wmnet with OS trixie
  • 14:19 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment
  • 14:19 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1050.eqiad.wmnet
  • 14:19 vgutierrez@cumin1004: END (ERROR) - Cookbook sre.hosts.powercycle (exit_code=97) for host lvs5006
  • 14:19 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1050.eqiad.wmnet
  • 14:14 vgutierrez@cumin1004: START - Cookbook sre.hosts.powercycle for host lvs5006
  • 14:14 dreamyjazz@deploy1003: dreamyjazz: Backport for AbuseReview: Enable queues on several wikis (T442476 T442204) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 14:13 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti1050.eqiad.wmnet
  • 14:11 vgutierrez@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host lvs5006.eqsin.wmnet with OS trixie
  • 14:11 vgutierrez@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs5006.eqsin.wmnet (remove config only)
  • 14:10 tchin@deploy1003: helmfile [staging] DONE helmfile.d/services/eventstreams: apply
  • 14:10 tchin@deploy1003: helmfile [staging] START helmfile.d/services/eventstreams: apply
  • 14:10 vgutierrez@cumin1004: START - Cookbook sre.hosts.dhcp for host lvs5006.eqsin.wmnet (remove config only)
  • 14:10 dreamyjazz@deploy1003: Started scap sync-world: Backport for AbuseReview: Enable queues on several wikis (T442476 T442204)
  • 14:09 vgutierrez@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs5006.eqsin.wmnet (push config only)
  • 14:09 vgutierrez@cumin1004: START - Cookbook sre.hosts.dhcp for host lvs5006.eqsin.wmnet (push config only)
  • 14:09 vgutierrez@cumin1004: START - Cookbook sre.hosts.reimage for host lvs5006.eqsin.wmnet with OS trixie
  • 14:04 tchin@deploy1003: helmfile [staging] DONE helmfile.d/services/eventstreams: apply
  • 14:03 vgutierrez@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host lvs5006.eqsin.wmnet with OS trixie
  • 14:03 vgutierrez@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs5006.eqsin.wmnet (remove config only)
  • 14:03 vgutierrez@cumin1004: START - Cookbook sre.hosts.dhcp for host lvs5006.eqsin.wmnet (remove config only)
  • 14:02 vgutierrez@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs5006.eqsin.wmnet (push config only)
  • 14:02 vgutierrez@cumin1004: START - Cookbook sre.hosts.dhcp for host lvs5006.eqsin.wmnet (push config only)
  • 14:02 tchin@deploy1003: helmfile [staging] START helmfile.d/services/eventstreams: apply
  • 14:01 vgutierrez@cumin1004: START - Cookbook sre.hosts.reimage for host lvs5006.eqsin.wmnet with OS trixie
  • 14:00 tchin@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/eventstreams-internal: apply
  • 13:59 tchin@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/eventstreams-internal: apply
  • 13:59 tchin@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/eventstreams-internal: apply
  • 13:56 moritzm: failover Ganeti master in esams to ganeti3005
  • 13:53 tchin@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/eventstreams-internal: apply
  • 13:52 tchin@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/eventstreams-internal: apply
  • 13:52 tchin@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/eventstreams-internal: apply
  • 13:52 akhatun@deploy1003: helmfile [eqiad] DONE helmfile.d/services/lac-lambdas: apply
  • 13:51 akhatun@deploy1003: helmfile [eqiad] START helmfile.d/services/lac-lambdas: apply
  • 13:51 akhatun@deploy1003: helmfile [codfw] DONE helmfile.d/services/lac-lambdas: apply
  • 13:51 dreamyjazz@deploy1003: Finished scap sync-world: Backport for AbuseReview: Allow promoting vandalism out of alpha test (T442379) (1.47.0-wmf.24, 1.48.0-wmf.1), WikimediaAntiAbuse: Promote vandalism queue out of alpha in enwiki (T442379) (duration: 34m 33s)
  • 13:49 akhatun@deploy1003: helmfile [codfw] START helmfile.d/services/lac-lambdas: apply
  • 13:47 akhatun@deploy1003: helmfile [staging] DONE helmfile.d/services/lac-lambdas: apply
  • 13:45 fabfur@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp3071.esams.wmnet
  • 13:45 akhatun@deploy1003: helmfile [staging] START helmfile.d/services/lac-lambdas: apply
  • 13:39 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment
  • 13:38 dreamyjazz@deploy1003: dreamyjazz: Backport for AbuseReview: Allow promoting vandalism out of alpha test (T442379) (1.47.0-wmf.24, 1.48.0-wmf.1), WikimediaAntiAbuse: Promote vandalism queue out of alpha in enwiki (T442379) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 13:35 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti3007.esams.wmnet
  • 13:35 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti3007.esams.wmnet
  • 13:33 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1050.eqiad.wmnet
  • 13:32 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1049.eqiad.wmnet
  • 13:32 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1049.eqiad.wmnet
  • 13:27 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti1049.eqiad.wmnet
  • 13:27 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti3007.esams.wmnet
  • 13:21 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti3007.esams.wmnet
  • 13:21 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1049.eqiad.wmnet
  • 13:16 dreamyjazz@deploy1003: Started scap sync-world: Backport for AbuseReview: Allow promoting vandalism out of alpha test (T442379) (1.47.0-wmf.24, 1.48.0-wmf.1), WikimediaAntiAbuse: Promote vandalism queue out of alpha in enwiki (T442379)
  • 13:15 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti3006.esams.wmnet
  • 13:15 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti3006.esams.wmnet
  • 13:15 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1038.eqiad.wmnet
  • 13:15 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1038.eqiad.wmnet
  • 13:10 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-fr-tech: apply
  • 13:09 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-fr-tech: apply
  • 13:07 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti3006.esams.wmnet
  • 13:07 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti1038.eqiad.wmnet
  • 13:04 fabfur@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp3070.esams.wmnet
  • 12:54 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti3006.esams.wmnet
  • 12:53 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1038.eqiad.wmnet
  • 12:38 fabfur@cumin1004: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on A:cp-upload_esams
  • 12:38 fabfur@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp3081.esams.wmnet
  • 12:30 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P{lvs4008.ulsfo.wmnet} and A:liberica
  • 12:30 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin config_reloading P{lvs4008.ulsfo.wmnet} and A:liberica
  • 12:29 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host build2002.codfw.wmnet
  • 12:28 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti3005.esams.wmnet
  • 12:28 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti3005.esams.wmnet
  • 12:27 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
  • 12:27 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'.
  • 12:26 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'.
  • 12:25 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host build2002.codfw.wmnet
  • 12:25 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'.
  • 12:24 cgoubert@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'.
  • 12:24 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1034.eqiad.wmnet
  • 12:24 cgoubert@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'.
  • 12:24 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1034.eqiad.wmnet
  • 12:22 cgoubert@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'.
  • 12:22 fabfur@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp3069.esams.wmnet
  • 12:21 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-master-codfw
  • 12:21 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl2006.codfw.wmnet
  • 12:21 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl2006.codfw.wmnet
  • 12:21 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast7002.wikimedia.org
  • 12:21 cgoubert@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'.
  • 12:20 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti3005.esams.wmnet
  • 12:20 herron@cumin1004: END (PASS) - Cookbook sre.opensearch.roll-restart-reboot (exit_code=0) rolling reboot on A:logstash-eqiad
  • 12:19 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti3005.esams.wmnet
  • 12:17 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti1034.eqiad.wmnet
  • 12:17 cgoubert@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'.
  • 12:15 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast7002.wikimedia.org
  • 12:15 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl2006.codfw.wmnet
  • 12:15 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl2006.codfw.wmnet
  • 12:15 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl2005.codfw.wmnet
  • 12:15 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl2005.codfw.wmnet
  • 12:14 cgoubert@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'.
  • 12:14 cgoubert@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'.
  • 12:13 cgoubert@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'.
  • 12:13 dbrant@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply
  • 12:13 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1034.eqiad.wmnet
  • 12:13 dbrant@deploy1003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply
  • 12:12 dbrant@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply
  • 12:12 dbrant@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply
  • 12:12 cgoubert@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'.
  • 12:11 cgoubert@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'.
  • 12:10 dbrant@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply
  • 12:10 dbrant@deploy1003: helmfile [staging] START helmfile.d/services/wikifeeds: apply
  • 12:09 cgoubert@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
  • 12:08 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl2005.codfw.wmnet
  • 12:08 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl2005.codfw.wmnet
  • 12:08 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl2004.codfw.wmnet
  • 12:08 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl2004.codfw.wmnet
  • 12:08 cgoubert@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
  • 12:08 cgoubert@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'.
  • 12:06 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply
  • 12:06 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1033.eqiad.wmnet
  • 12:06 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1033.eqiad.wmnet
  • 12:06 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply
  • 12:06 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply
  • 12:06 blake@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply
  • 12:06 blake@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply
  • 12:05 cgoubert@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'.
  • 12:05 blake@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply
  • 12:02 vgutierrez@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host lvs4008.ulsfo.wmnet with OS trixie
  • 12:02 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl2004.codfw.wmnet
  • 12:02 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl2004.codfw.wmnet
  • 12:02 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl2003.codfw.wmnet
  • 12:02 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl2003.codfw.wmnet
  • 12:00 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti1033.eqiad.wmnet
  • 12:00 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
  • 12:00 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
  • 12:00 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
  • 12:00 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: sync
  • 11:57 fabfur@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp3080.esams.wmnet
  • 11:55 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl2003.codfw.wmnet
  • 11:54 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl2003.codfw.wmnet
  • 11:54 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl2002.codfw.wmnet
  • 11:54 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl2002.codfw.wmnet
  • 11:54 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply
  • 11:54 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply
  • 11:54 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply
  • 11:54 blake@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply
  • 11:54 blake@deploy1003: Finished scap sync-world: Backport for GrowthExperiments: Set link recommendation service timeout to 32s (T442468) (duration: 10m 18s)
  • 11:48 blake@deploy1003: urbanecm, blake: Continuing with deployment
  • 11:47 blake@deploy1003: urbanecm, blake: Backport for GrowthExperiments: Set link recommendation service timeout to 32s (T442468) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 11:47 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl2002.codfw.wmnet
  • 11:47 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl2002.codfw.wmnet
  • 11:47 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl2001.codfw.wmnet
  • 11:47 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl2001.codfw.wmnet
  • 11:43 blake@deploy1003: Started scap sync-world: Backport for GrowthExperiments: Set link recommendation service timeout to 32s (T442468)
  • 11:43 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1033.eqiad.wmnet
  • 11:42 vgutierrez@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs4008.ulsfo.wmnet with reason: host reimage
  • 11:40 fabfur@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp3068.esams.wmnet
  • 11:39 vgutierrez@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on lvs4008.ulsfo.wmnet with reason: host reimage
  • 11:37 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl2001.codfw.wmnet
  • 11:37 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl2001.codfw.wmnet
  • 11:37 cgoubert@cumin2003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-master-codfw
  • 11:37 vgutierrez@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs4008.ulsfo.wmnet (remove config only)
  • 11:36 vgutierrez@cumin1004: START - Cookbook sre.hosts.dhcp for host lvs4008.ulsfo.wmnet (remove config only)
  • 11:33 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rdb1015.eqiad.wmnet
  • 11:32 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rdb2013.codfw.wmnet
  • 11:30 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest2006.codfw.wmnet with OS trixie
  • 11:30 vgutierrez: testing purged 0.25+deb13u2 in cp7010 and cp7016 - T442513
  • 11:28 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host rdb1015.eqiad.wmnet
  • 11:28 vgutierrez: upload purged 0.25+deb13u2 to trixie-wikimedia (apt.wm.o) - T442513
  • 11:26 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host rdb2013.codfw.wmnet
  • 11:23 vgutierrez@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs4008.ulsfo.wmnet (push config only)
  • 11:23 vgutierrez@cumin1004: START - Cookbook sre.hosts.dhcp for host lvs4008.ulsfo.wmnet (push config only)
  • 11:22 vgutierrez@cumin1004: START - Cookbook sre.hosts.reimage for host lvs4008.ulsfo.wmnet with OS trixie
  • 11:19 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rdb2011.codfw.wmnet
  • 11:19 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rdb1013.eqiad.wmnet
  • 11:16 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P{lvs4008.ulsfo.wmnet} and A:liberica
  • 11:15 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin config_reloading P{lvs4008.ulsfo.wmnet} and A:liberica
  • 11:14 fabfur@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp3079.esams.wmnet
  • 11:14 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host rdb2011.codfw.wmnet
  • 11:13 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2006.codfw.wmnet with reason: host reimage
  • 11:12 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host rdb1013.eqiad.wmnet
  • 11:09 elukey@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2006.codfw.wmnet with reason: host reimage
  • 11:07 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2006.codfw.wmnet (remove config only)
  • 11:07 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest2006.codfw.wmnet (remove config only)
  • 11:03 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rdb-lock1003.eqiad.wmnet
  • 11:00 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P{lvs4009.ulsfo.wmnet} and A:liberica
  • 10:59 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin config_reloading P{lvs4009.ulsfo.wmnet} and A:liberica
  • 10:59 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host rdb-lock1003.eqiad.wmnet
  • 10:58 fabfur@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp3067.esams.wmnet
  • 10:57 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rdb-lock1002.eqiad.wmnet
  • 10:57 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rdb2014.codfw.wmnet
  • 10:56 dpogorzelski@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply
  • 10:56 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2006.codfw.wmnet (push config only)
  • 10:56 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest2006.codfw.wmnet (push config only)
  • 10:56 elukey@cumin1004: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS trixie
  • 10:55 dpogorzelski@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply
  • 10:54 dpogorzelski@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply
  • 10:54 dpogorzelski@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply
  • 10:54 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host rdb-lock1002.eqiad.wmnet
  • 10:52 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rdb-lock1001.eqiad.wmnet
  • 10:51 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host rdb2014.codfw.wmnet
  • 10:51 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rdb2012.codfw.wmnet
  • 10:48 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host rdb-lock1001.eqiad.wmnet
  • 10:47 dpogorzelski@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply
  • 10:46 dpogorzelski@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply
  • 10:46 slyngshede@cumin1004: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P{cp2049*} OR P{cp2051*} OR P{cp2053*} OR P{cp2055*} OR P{cp2057*} OR P{cp2059*} and A:cp
  • 10:46 slyngshede@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp2059.codfw.wmnet
  • 10:45 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host rdb2012.codfw.wmnet
  • 10:44 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rdb1016.eqiad.wmnet
  • 10:42 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rdb-lock2003.codfw.wmnet
  • 10:38 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host rdb1016.eqiad.wmnet
  • 10:38 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host rdb-lock2003.codfw.wmnet
  • 10:36 vgutierrez@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host lvs4009.ulsfo.wmnet with OS trixie
  • 10:32 fabfur@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp3078.esams.wmnet
  • 10:31 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rdb-lock2002.codfw.wmnet
  • 10:29 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rdb1014.eqiad.wmnet
  • 10:27 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host rdb-lock2002.codfw.wmnet
  • 10:25 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rdb-lock2001.codfw.wmnet
  • 10:23 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host rdb1014.eqiad.wmnet
  • 10:22 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply
  • 10:22 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host rdb-lock2001.codfw.wmnet
  • 10:21 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply
  • 10:21 cgoubert@cumin2003: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host rdb-lock2001.codfw.wmnet
  • 10:21 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host rdb-lock2001.codfw.wmnet
  • 10:20 cgoubert@cumin2003: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host rdb-lock2001.codfw.wmnet
  • 10:20 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host rdb-lock2001.codfw.wmnet
  • 10:17 vgutierrez@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs4009.ulsfo.wmnet with reason: host reimage
  • 10:16 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1058.eqiad.wmnet
  • 10:16 fabfur@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp3066.esams.wmnet
  • 10:16 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1058.eqiad.wmnet
  • 10:14 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2006.codfw.wmnet (remove config only)
  • 10:14 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest2006.codfw.wmnet (remove config only)
  • 10:13 vgutierrez@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on lvs4009.ulsfo.wmnet with reason: host reimage
  • 10:11 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti1058.eqiad.wmnet
  • 10:11 vgutierrez@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs4009.ulsfo.wmnet (remove config only)
  • 10:10 vgutierrez@cumin1004: START - Cookbook sre.hosts.dhcp for host lvs4009.ulsfo.wmnet (remove config only)
  • 10:08 slyngshede@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp2057.codfw.wmnet
  • 10:06 fabfur@cumin1004: START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-text_esams
  • 10:02 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2044.codfw.wmnet
  • 10:02 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2044.codfw.wmnet
  • 09:58 vgutierrez@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs4009.ulsfo.wmnet (push config only)
  • 09:57 vgutierrez@cumin1004: START - Cookbook sre.hosts.dhcp for host lvs4009.ulsfo.wmnet (push config only)
  • 09:56 vgutierrez@cumin1004: START - Cookbook sre.hosts.reimage for host lvs4009.ulsfo.wmnet with OS trixie
  • 09:56 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1058.eqiad.wmnet
  • 09:56 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti2044.codfw.wmnet
  • 09:53 fabfur@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp3077.esams.wmnet
  • 09:52 elukey@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2006.codfw.wmnet with OS trixie
  • 09:51 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1048.eqiad.wmnet
  • 09:51 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1048.eqiad.wmnet
  • 09:49 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P{lvs4009.ulsfo.wmnet} and A:liberica
  • 09:49 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin config_reloading P{lvs4009.ulsfo.wmnet} and A:liberica
  • 09:49 godog: extend silence ATSBackendErrorsHigh eventstreams.discovery.wmnet to all sites for another ~8h - T442462
  • 09:48 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P{lvs4009.ulsfo.wmnet} and A:liberica
  • 09:48 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin config_reloading P{lvs4009.ulsfo.wmnet} and A:liberica
  • 09:46 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti1048.eqiad.wmnet
  • 09:46 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2044.codfw.wmnet
  • 09:45 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2043.codfw.wmnet
  • 09:45 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2043.codfw.wmnet
  • 09:41 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2006.codfw.wmnet (push config only)
  • 09:41 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest2006.codfw.wmnet (push config only)
  • 09:41 elukey@cumin1004: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS trixie
  • 09:39 elukey@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2006.codfw.wmnet with OS trixie
  • 09:39 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet
  • 09:39 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti2043.codfw.wmnet
  • 09:37 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2006.codfw.wmnet (push config only)
  • 09:37 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest2006.codfw.wmnet (push config only)
  • 09:37 elukey@cumin1004: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS trixie
  • 09:36 elukey@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host ml-serve1016.eqiad.wmnet with OS trixie
  • 09:36 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host ml-serve1016.eqiad.wmnet (remove config only)
  • 09:36 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host ml-serve1016.eqiad.wmnet (remove config only)
  • 09:35 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1047.eqiad.wmnet
  • 09:34 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1047.eqiad.wmnet
  • 09:29 slyngshede@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp2055.codfw.wmnet
  • 09:29 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti1047.eqiad.wmnet
  • 09:27 elukey@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2006.codfw.wmnet with OS trixie
  • 09:25 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2043.codfw.wmnet
  • 09:24 moritzm: installing ca-certificates updates for trixie
  • 09:24 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2042.codfw.wmnet
  • 09:23 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2042.codfw.wmnet
  • 09:18 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1047.eqiad.wmnet
  • 09:18 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti2042.codfw.wmnet
  • 09:17 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host ml-serve1016.eqiad.wmnet (push config only)
  • 09:17 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host ml-serve1016.eqiad.wmnet (push config only)
  • 09:17 elukey@cumin1004: START - Cookbook sre.hosts.reimage for host ml-serve1016.eqiad.wmnet with OS trixie
  • 09:16 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2006.codfw.wmnet (push config only)
  • 09:16 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest2006.codfw.wmnet (push config only)
  • 09:16 elukey@cumin1004: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS trixie
  • 09:15 vgutierrez@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host lvs4010.ulsfo.wmnet with OS trixie
  • 09:15 elukey@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2006.codfw.wmnet with OS trixie
  • 09:11 fabfur@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp3076.esams.wmnet
  • 09:08 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1045.eqiad.wmnet
  • 09:08 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1045.eqiad.wmnet
  • 09:07 tappof: bump space for prometheus k8s-(dse|aux|ml-serve) in eqiad
  • 09:05 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2006.codfw.wmnet (push config only)
  • 09:05 hashar@deploy1003: Finished scap sync-world: Backport for Allow those with CheckUser right to view private filters on WMF projects (T442464) (duration: 15m 12s)
  • 09:05 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest2006.codfw.wmnet (push config only)
  • 09:05 elukey@cumin1004: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS trixie
  • 09:04 moritzm: upgrade rpki2003 to routinator 0.15.3
  • 09:02 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti1045.eqiad.wmnet
  • 09:02 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2042.codfw.wmnet
  • 09:02 moritzm: imported routinator 0.15.3 to thirdparty/routinator
  • 08:59 fabfur@cumin1004: START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-upload_esams
  • 08:59 hashar@deploy1003: neriah, hashar: Continuing with deployment
  • 08:58 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2006.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 08:55 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.powercycle (exit_code=0) for host ml-serve1016
  • 08:54 vgutierrez@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs4010.ulsfo.wmnet with reason: host reimage
  • 08:54 hashar@deploy1003: neriah, hashar: Backport for Allow those with CheckUser right to view private filters on WMF projects (T442464) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 08:51 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2041.codfw.wmnet
  • 08:51 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2041.codfw.wmnet
  • 08:50 slyngshede@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp2053.codfw.wmnet
  • 08:50 elukey@cumin1004: START - Cookbook sre.hosts.powercycle for host ml-serve1016
  • 08:49 hashar@deploy1003: Started scap sync-world: Backport for Allow those with CheckUser right to view private filters on WMF projects (T442464)
  • 08:49 vgutierrez@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on lvs4010.ulsfo.wmnet with reason: host reimage
  • 08:48 elukey@cumin1004: START - Cookbook sre.hosts.provision for host sretest2006.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 08:48 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1045.eqiad.wmnet
  • 08:46 vgutierrez@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs4010.ulsfo.wmnet (remove config only)
  • 08:46 vgutierrez@cumin1004: START - Cookbook sre.hosts.dhcp for host lvs4010.ulsfo.wmnet (remove config only)
  • 08:46 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti2041.codfw.wmnet
  • 08:45 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1037.eqiad.wmnet
  • 08:45 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1037.eqiad.wmnet
  • 08:43 elukey@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['sretest2006']
  • 08:40 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.powercycle (exit_code=0) for host ml-serve1016
  • 08:40 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2041.codfw.wmnet
  • 08:40 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2026.codfw.wmnet
  • 08:39 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2026.codfw.wmnet
  • 08:37 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti1037.eqiad.wmnet
  • 08:35 elukey@cumin1004: START - Cookbook sre.hosts.powercycle for host ml-serve1016
  • 08:35 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1037.eqiad.wmnet
  • 08:35 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti2026.codfw.wmnet
  • 08:35 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1028.eqiad.wmnet
  • 08:34 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1028.eqiad.wmnet
  • 08:33 vgutierrez@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs4010.ulsfo.wmnet (push config only)
  • 08:33 vgutierrez@cumin1004: START - Cookbook sre.hosts.dhcp for host lvs4010.ulsfo.wmnet (push config only)
  • 08:32 vgutierrez@cumin1004: START - Cookbook sre.hosts.reimage for host lvs4010.ulsfo.wmnet with OS trixie
  • 08:29 elukey@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host ml-serve1016.eqiad.wmnet with OS trixie
  • 08:29 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host ml-serve1016.eqiad.wmnet (remove config only)
  • 08:29 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host ml-serve1016.eqiad.wmnet (remove config only)
  • 08:28 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti1028.eqiad.wmnet
  • 08:28 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2026.codfw.wmnet
  • 08:26 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2025.codfw.wmnet
  • 08:26 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2025.codfw.wmnet
  • 08:20 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti2025.codfw.wmnet
  • 08:20 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1028.eqiad.wmnet
  • 08:19 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1027.eqiad.wmnet
  • 08:19 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1027.eqiad.wmnet
  • 08:15 marostegui: Enabled database circular replication https://phabricator.wikimedia.org/T439063
  • 08:15 marostegui@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section s1
  • 08:13 slyngshede@cumin1004: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P{cp2056*} OR P{cp2058*} and A:cp
  • 08:13 slyngshede@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp2058.codfw.wmnet
  • 08:13 marostegui@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section s1
  • 08:13 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti1027.eqiad.wmnet
  • 08:11 slyngshede@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp2051.codfw.wmnet
  • 08:10 elukey@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['sretest2006']
  • 08:09 elukey@cumin1004: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['sretest2006']
  • 08:09 marostegui@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section s4
  • 08:09 elukey@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['sretest2006']
  • 08:09 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2025.codfw.wmnet
  • 08:08 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1027.eqiad.wmnet
  • 08:08 marostegui@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section s4
  • 08:07 marostegui@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section s8
  • 08:06 marostegui@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section s8
  • 08:06 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host ml-serve1016.eqiad.wmnet (push config only)
  • 08:05 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host ml-serve1016.eqiad.wmnet (push config only)
  • 08:05 marostegui@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section s7
  • 08:05 elukey@cumin1004: START - Cookbook sre.hosts.reimage for host ml-serve1016.eqiad.wmnet with OS trixie
  • 08:04 marostegui@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section s7
  • 08:03 marostegui@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section s3
  • 08:02 marostegui@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section s3
  • 07:52 marostegui@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section s2
  • 07:50 marostegui@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section s2
  • 07:38 arthurtaylor@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply
  • 07:38 arthurtaylor@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply
  • 07:38 arthurtaylor@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikidata-query-gui: apply
  • 07:38 arthurtaylor@deploy1003: helmfile [codfw] START helmfile.d/services/wikidata-query-gui: apply
  • 07:37 arthurtaylor@deploy1003: helmfile [staging] DONE helmfile.d/services/wikidata-query-gui: apply
  • 07:37 arthurtaylor@deploy1003: helmfile [staging] START helmfile.d/services/wikidata-query-gui: apply
  • 07:35 slyngshede@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp2056.codfw.wmnet
  • 07:32 slyngshede@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp2049.codfw.wmnet
  • 07:26 marostegui@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section s5
  • 07:24 marostegui@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section s5
  • 07:24 slyngshede@cumin1004: START - Cookbook sre.cdn.roll-reboot rolling reboot on P{cp2056*} OR P{cp2058*} and A:cp
  • 07:24 slyngshede@cumin1004: START - Cookbook sre.cdn.roll-reboot rolling reboot on P{cp2049*} OR P{cp2051*} OR P{cp2053*} OR P{cp2055*} OR P{cp2057*} OR P{cp2059*} and A:cp
  • 07:23 slyngshede@cumin1004: END (ERROR) - Cookbook sre.cdn.roll-reboot (exit_code=97) rolling reboot on P{cp2049*} OR P{cp2051*} OR P{cp2053*} OR P{cp2055*} OR P{cp2057*} OR P{cp2059*} and A:cp
  • 07:23 slyngshede@cumin1004: START - Cookbook sre.cdn.roll-reboot rolling reboot on P{cp2049*} OR P{cp2051*} OR P{cp2053*} OR P{cp2055*} OR P{cp2057*} OR P{cp2059*} and A:cp
  • 07:23 marostegui@cumin1004: END (FAIL) - Cookbook sre.switchdc.databases.prepare (exit_code=93) for the switch from eqiad to codfw for section s5
  • 07:23 marostegui@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section s5
  • 07:18 marostegui@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section s6
  • 07:17 marostegui@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section s6
  • 07:10 marostegui@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section es7
  • 07:09 marostegui@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section es7
  • 07:09 marostegui@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section es6
  • 07:07 marostegui@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section es6
  • 06:49 godog: add 5xx responses as a % of total traffic to https://grafana.wikimedia.org/d/1T_4O08Wk/ats-backends-3c-3e-origin-servers-overview - T400675
  • 06:37 marostegui@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section x4
  • 06:35 marostegui@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section x4
  • 06:29 hashar@deploy1003: Finished deploy [gerrit/gerrit@38dd63c]: Disable banner about emergency maintenance (duration: 00m 14s)
  • 06:29 hashar@deploy1003: Started deploy [gerrit/gerrit@38dd63c]: Disable banner about emergency maintenance
  • 06:18 godog: silence ATSBackendErrorsHigh eventstreams.discovery.wmnet drmrs page for 12h - T442462
  • 05:19 tchin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/eventstreams: apply
  • 05:19 tchin@deploy1003: helmfile [eqiad] START helmfile.d/services/eventstreams: apply
  • 00:00 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
  • 00:00 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: sync

2026-10-26

  • {{safesubst:SAL entry|1=23:21 jdlrobson@deploy1003: Finished scap sync-world: Backport for ReadingList: Fix instrument for clicking on overflow icon (T441477) (1.47.0-wmf.24, 1.48.0-wmf.1), Fix recentlyDonated captures non-donors (T441244) (1.48.0-wmf.1, 1.47.0-wmf.24), Update account creation instrumentation mechanism (T429288) (1.48.0-wmf.1, [[gerrit:1}}
  • 23:13 jdlrobson@deploy1003: jdlrobson: Continuing with deployment
  • {{safesubst:SAL entry|1=23:09 jdlrobson@deploy1003: jdlrobson: Backport for ReadingList: Fix instrument for clicking on overflow icon (T441477) (1.47.0-wmf.24, 1.48.0-wmf.1), Fix recentlyDonated captures non-donors (T441244) (1.48.0-wmf.1, 1.47.0-wmf.24), Update account creation instrumentation mechanism (T429288) (1.48.0-wmf.1, [[gerrit:1358786|1.47.0-w}}
  • 23:06 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-worker-codfw
  • 23:06 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker-exp2001.codfw.wmnet
  • 23:06 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker-exp2001.codfw.wmnet
  • {{safesubst:SAL entry|1=23:04 jdlrobson@deploy1003: Started scap sync-world: Backport for ReadingList: Fix instrument for clicking on overflow icon (T441477) (1.47.0-wmf.24, 1.48.0-wmf.1), Fix recentlyDonated captures non-donors (T441244) (1.48.0-wmf.1, 1.47.0-wmf.24), Update account creation instrumentation mechanism (T429288) (1.48.0-wmf.1, [[gerrit:13}}
  • 23:03 brett@cumin2003: END (PASS) - Cookbook sre.cdn.roll-upgrade-ats (exit_code=0) Rolling upgrade of ATS on A:cp-text_esams and A:cp - 9.2.16 upgrade (T441945)
  • 23:02 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker-exp2001.codfw.wmnet
  • 23:01 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker-exp2001.codfw.wmnet
  • 23:01 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2361-2374].codfw.wmnet
  • 23:01 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2361-2374].codfw.wmnet
  • 23:01 sbassett: Deployed security fixes for T440493 and T441721 to wmf.24 and wmf.1
  • 22:54 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2361-2374].codfw.wmnet
  • 22:46 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2361-2374].codfw.wmnet
  • 22:45 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2347-2360].codfw.wmnet
  • 22:45 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2347-2360].codfw.wmnet
  • 22:38 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2347-2360].codfw.wmnet
  • 22:38 brett@cumin2003: END (PASS) - Cookbook sre.cdn.roll-upgrade-ats (exit_code=0) Rolling upgrade of ATS on A:cp-upload_esams and A:cp - 9.2.16 upgrade (T441945)
  • 22:24 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2347-2360].codfw.wmnet
  • 22:24 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2333-2346].codfw.wmnet
  • 22:24 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2333-2346].codfw.wmnet
  • 22:16 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2333-2346].codfw.wmnet
  • 22:08 sbassett: Deployed security fix for T440638 to wmf.24 and wmf.1
  • 22:08 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2333-2346].codfw.wmnet
  • 22:08 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2319-2332].codfw.wmnet
  • 22:07 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2319-2332].codfw.wmnet
  • 22:03 brett@cumin2003: START - Cookbook sre.cdn.roll-upgrade-ats Rolling upgrade of ATS on A:cp-upload_esams and A:cp - 9.2.16 upgrade (T441945)
  • 22:03 brett@cumin2003: START - Cookbook sre.cdn.roll-upgrade-ats Rolling upgrade of ATS on A:cp-text_esams and A:cp - 9.2.16 upgrade (T441945)
  • 22:02 brett@cumin2003: END (PASS) - Cookbook sre.cdn.roll-upgrade-ats (exit_code=0) Rolling upgrade of ATS on A:cp-text_drmrs and A:cp - 9.2.16 upgrade (T441945)
  • 22:00 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2319-2332].codfw.wmnet
  • 21:52 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2319-2332].codfw.wmnet
  • 21:52 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2305-2318].codfw.wmnet
  • 21:52 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2305-2318].codfw.wmnet
  • 21:49 sbassett: Deployed security fix for T442018 to wmf.24 and wmf.1
  • 21:44 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2305-2318].codfw.wmnet
  • 21:36 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2305-2318].codfw.wmnet
  • 21:36 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2291-2304].codfw.wmnet
  • 21:35 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2291-2304].codfw.wmnet
  • 21:33 brett@cumin2003: END (PASS) - Cookbook sre.cdn.roll-upgrade-ats (exit_code=0) Rolling upgrade of ATS on A:cp-upload_drmrs and A:cp - 9.2.16 upgrade (T441945)
  • 21:28 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2291-2304].codfw.wmnet
  • 21:23 sbassett: Deployed security fix for T442373 and T441029 to wmf.24 and wmf.1
  • 21:16 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2291-2304].codfw.wmnet
  • 21:15 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2277-2290].codfw.wmnet
  • 21:15 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2277-2290].codfw.wmnet
  • 21:06 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2277-2290].codfw.wmnet
  • 20:58 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2277-2290].codfw.wmnet
  • 20:58 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2263-2276].codfw.wmnet
  • 20:58 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2263-2276].codfw.wmnet
  • 20:56 brett@cumin2003: START - Cookbook sre.cdn.roll-upgrade-ats Rolling upgrade of ATS on A:cp-text_drmrs and A:cp - 9.2.16 upgrade (T441945)
  • 20:56 brett@cumin2003: START - Cookbook sre.cdn.roll-upgrade-ats Rolling upgrade of ATS on A:cp-upload_drmrs and A:cp - 9.2.16 upgrade (T441945)
  • 20:51 brett@cumin2003: END (PASS) - Cookbook sre.cdn.roll-upgrade-ats (exit_code=0) Rolling upgrade of ATS on A:cp-text_ulsfo and not P{cp4037.ulsfo.wmnet} and A:cp - 9.2.16 upgrade (T441945)
  • 20:50 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2263-2276].codfw.wmnet
  • 20:42 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2263-2276].codfw.wmnet
  • 20:41 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2249-2262].codfw.wmnet
  • 20:41 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2249-2262].codfw.wmnet
  • 20:33 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2249-2262].codfw.wmnet
  • 20:32 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply
  • 20:32 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply
  • 20:30 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
  • 20:30 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
  • 20:29 urbanecm@deploy1003: mwscript-k8s job started: GrowthExperiments:revalidateLinkRecommendations.php --wiki=enwiki --olderThan=1790175600 --verbose # T438366
  • 20:27 brett@cumin2003: END (PASS) - Cookbook sre.cdn.roll-upgrade-ats (exit_code=0) Rolling upgrade of ATS on A:cp-upload_ulsfo and not P{cp4052.ulsfo.wmnet} and A:cp - 9.2.16 upgrade (T441945)
  • 20:27 urbanecm@deploy1003: mwscript-k8s job started: foreachwikiindblist mwscript.dblist GrowthExperiments:revalidateLinkRecommendations.php --olderThan=1790175600 --verbose # T438366 (dblist: https://phabricator.wikimedia.org/P97002)
  • 20:25 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2249-2262].codfw.wmnet
  • 20:25 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
  • 20:24 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2205-2215,2242-2243,2248].codfw.wmnet
  • 20:24 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2205-2215,2242-2243,2248].codfw.wmnet
  • 20:23 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply
  • 20:21 urbanecm@deploy1003: Finished scap sync-world: Backport for Revert "Improve types" (T441875) (1.48.0-wmf.1) (duration: 10m 30s)
  • 20:21 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
  • 20:20 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply
  • 20:16 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2205-2215,2242-2243,2248].codfw.wmnet
  • 20:16 urbanecm@deploy1003: somerandomdeveloper, urbanecm: Continuing with deployment
  • 20:15 urbanecm@deploy1003: somerandomdeveloper, urbanecm: Backport for Revert "Improve types" (T441875) (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 20:11 urbanecm@deploy1003: Started scap sync-world: Backport for Revert "Improve types" (T441875) (1.48.0-wmf.1)
  • 20:07 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2205-2215,2242-2243,2248].codfw.wmnet
  • 20:07 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2191-2204].codfw.wmnet
  • 20:07 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2191-2204].codfw.wmnet
  • 20:02 brett@cumin2003: START - Cookbook sre.cdn.roll-upgrade-ats Rolling upgrade of ATS on A:cp-upload_ulsfo and not P{cp4052.ulsfo.wmnet} and A:cp - 9.2.16 upgrade (T441945)
  • 20:01 brett@cumin2003: START - Cookbook sre.cdn.roll-upgrade-ats Rolling upgrade of ATS on A:cp-text_ulsfo and not P{cp4037.ulsfo.wmnet} and A:cp - 9.2.16 upgrade (T441945)
  • 19:58 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2191-2204].codfw.wmnet
  • 19:49 brett@cumin2003: END (PASS) - Cookbook sre.cdn.roll-upgrade-ats (exit_code=0) Rolling upgrade of ATS on P{cp4037.ulsfo.wmnet} and A:cp - 9.2.16 upgrade (T441945)
  • 19:49 brett@cumin2003: END (PASS) - Cookbook sre.cdn.roll-upgrade-ats (exit_code=0) Rolling upgrade of ATS on P{cp4052.ulsfo.wmnet} and A:cp - 9.2.16 upgrade (T441945)
  • 19:46 jhathaway@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on mx-in2001.wikimedia.org with reason: new-kernel
  • 19:45 brett@cumin2003: START - Cookbook sre.cdn.roll-upgrade-ats Rolling upgrade of ATS on P{cp4037.ulsfo.wmnet} and A:cp - 9.2.16 upgrade (T441945)
  • 19:44 brett@cumin2003: START - Cookbook sre.cdn.roll-upgrade-ats Rolling upgrade of ATS on P{cp4052.ulsfo.wmnet} and A:cp - 9.2.16 upgrade (T441945)
  • 19:44 jhathaway@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on mx-in1001.wikimedia.org with reason: new-kernel
  • 19:43 krinkle@deploy1003: Finished scap sync-world: Backport for Remove unused StatsD configuration and instruments (T228380), labs: Remove redundant wgSkipSkins override (duration: 16m 48s)
  • 19:41 jhathaway@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on mx-out1001.wikimedia.org with reason: new-kernel
  • 19:41 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2191-2204].codfw.wmnet
  • 19:40 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2173-2179,2184-2190].codfw.wmnet
  • 19:40 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2173-2179,2184-2190].codfw.wmnet
  • 19:37 krinkle@deploy1003: krinkle, awight: Continuing with deployment
  • 19:35 jhathaway@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on mx-out2001.wikimedia.org with reason: new-kernel
  • 19:32 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2173-2179,2184-2190].codfw.wmnet
  • 19:30 krinkle@deploy1003: krinkle, awight: Backport for Remove unused StatsD configuration and instruments (T228380), labs: Remove redundant wgSkipSkins override synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 19:26 krinkle@deploy1003: Started scap sync-world: Backport for Remove unused StatsD configuration and instruments (T228380), labs: Remove redundant wgSkipSkins override
  • 19:25 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply
  • 19:25 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply
  • 19:24 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2173-2179,2184-2190].codfw.wmnet
  • 19:23 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2158-2172].codfw.wmnet
  • 19:23 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2158-2172].codfw.wmnet
  • 19:14 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2158-2172].codfw.wmnet
  • 19:07 akhatun@deploy1003: Finished scap sync-world: Backport for Prefetch the LAC revision_feature_counts artifact on testwiki (T436695) (duration: 25m 40s)
  • 19:06 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2158-2172].codfw.wmnet
  • 19:06 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2143-2157].codfw.wmnet
  • 19:05 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2143-2157].codfw.wmnet
  • 19:05 brett@cumin2003: END (PASS) - Cookbook sre.cdn.roll-upgrade-ats (exit_code=0) Rolling upgrade of ATS on A:cp-text_eqsin and not P{cp5001.eqsin.wmnet} and A:cp - 9.2.16 upgrade (T441945)
  • 19:05 jgreen@dns2006: END - running authdns-update
  • 19:03 jgreen@dns2006: START - running authdns-update
  • 19:01 akhatun@deploy1003: akhatun, otto: Continuing with deployment
  • 18:57 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2143-2157].codfw.wmnet
  • 18:48 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2143-2157].codfw.wmnet
  • 18:48 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2128-2142].codfw.wmnet
  • 18:48 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2128-2142].codfw.wmnet
  • 18:46 akhatun@deploy1003: akhatun, otto: Backport for Prefetch the LAC revision_feature_counts artifact on testwiki (T436695) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 18:41 akhatun@deploy1003: Started scap sync-world: Backport for Prefetch the LAC revision_feature_counts artifact on testwiki (T436695)
  • 18:39 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2128-2142].codfw.wmnet
  • 18:34 brett@cumin2003: END (PASS) - Cookbook sre.cdn.roll-upgrade-ats (exit_code=0) Rolling upgrade of ATS on A:cp-upload_eqsin and not P{cp5017.eqsin.wmnet} and A:cp - 9.2.16 upgrade (T441945)
  • 18:32 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply
  • 18:31 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply
  • 18:31 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2128-2142].codfw.wmnet
  • 18:31 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2105-2115,2124-2127].codfw.wmnet
  • 18:31 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2105-2115,2124-2127].codfw.wmnet
  • 18:30 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply
  • 18:30 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply
  • 18:22 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2105-2115,2124-2127].codfw.wmnet
  • 18:12 jiji@cumin2003: END (PASS) - Cookbook sre.memcached.roll-reboot-restart (exit_code=0) rolling reboot on A:memcached-codfw
  • 18:10 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2105-2115,2124-2127].codfw.wmnet
  • 18:09 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2076-2078,2087-2095,2102-2104].codfw.wmnet
  • 18:09 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2076-2078,2087-2095,2102-2104].codfw.wmnet
  • 18:04 aokoth@cumin1004: END (PASS) - Cookbook sre.vrts.upgrade (exit_code=0) on VRTS host vrts1003.eqiad.wmnet
  • 18:02 aokoth@cumin1004: START - Cookbook sre.vrts.upgrade on VRTS host vrts1003.eqiad.wmnet
  • 18:00 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2076-2078,2087-2095,2102-2104].codfw.wmnet
  • 17:53 brett@cumin2003: START - Cookbook sre.cdn.roll-upgrade-ats Rolling upgrade of ATS on A:cp-upload_eqsin and not P{cp5017.eqsin.wmnet} and A:cp - 9.2.16 upgrade (T441945)
  • 17:53 brett@cumin2003: START - Cookbook sre.cdn.roll-upgrade-ats Rolling upgrade of ATS on A:cp-text_eqsin and not P{cp5001.eqsin.wmnet} and A:cp - 9.2.16 upgrade (T441945)
  • 17:51 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2076-2078,2087-2095,2102-2104].codfw.wmnet
  • 17:50 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2059-2062,2064-2065,2067-2075].codfw.wmnet
  • 17:50 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2059-2062,2064-2065,2067-2075].codfw.wmnet
  • 17:50 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply
  • 17:49 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply
  • 17:48 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply
  • 17:48 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply
  • 17:42 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2040.codfw.wmnet
  • 17:42 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2040.codfw.wmnet
  • 17:42 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2059-2062,2064-2065,2067-2075].codfw.wmnet
  • 17:37 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti2040.codfw.wmnet
  • 17:36 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply
  • 17:35 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply
  • 17:34 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2059-2062,2064-2065,2067-2075].codfw.wmnet
  • 17:33 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2036-2039,2041-2042,2044,2046,2049-2051,2055-2058].codfw.wmnet
  • 17:33 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2036-2039,2041-2042,2044,2046,2049-2051,2055-2058].codfw.wmnet
  • 17:30 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply
  • 17:29 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply
  • 17:28 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply
  • 17:26 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply
  • 17:26 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
  • 17:26 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
  • 17:25 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
  • 17:25 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
  • 17:24 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2036-2039,2041-2042,2044,2046,2049-2051,2055-2058].codfw.wmnet
  • 17:21 dancy@deploy1003: Installation of scap version "4.306.0" completed for 3 hosts
  • 17:19 dancy@deploy1003: Installing scap version "4.306.0" for 3 host(s)
  • 17:16 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P{lvs7001.magru.wmnet} and A:liberica
  • 17:16 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2036-2039,2041-2042,2044,2046,2049-2051,2055-2058].codfw.wmnet
  • 17:15 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin config_reloading P{lvs7001.magru.wmnet} and A:liberica
  • 17:15 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2001-2002,2005-2006,2011-2018,2033-2035].codfw.wmnet
  • 17:15 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2001-2002,2005-2006,2011-2018,2033-2035].codfw.wmnet
  • 17:15 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1044.eqiad.wmnet
  • 17:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1044.eqiad.wmnet
  • 17:09 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti1044.eqiad.wmnet
  • 17:09 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2040.codfw.wmnet
  • 17:07 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2039.codfw.wmnet
  • 17:07 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2039.codfw.wmnet
  • 17:06 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2001-2002,2005-2006,2011-2018,2033-2035].codfw.wmnet
  • 17:04 vgutierrez@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host lvs7001.magru.wmnet with OS trixie
  • 17:01 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1044.eqiad.wmnet
  • 17:01 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti2039.codfw.wmnet
  • 16:53 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2001-2002,2005-2006,2011-2018,2033-2035].codfw.wmnet
  • 16:53 jasmine@cumin2003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-worker-codfw
  • 16:44 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1043.eqiad.wmnet
  • 16:44 vgutierrez@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs7001.magru.wmnet with reason: host reimage
  • 16:43 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1043.eqiad.wmnet
  • 16:41 atsukoito: upgrading airflow-dumps to airflow 3.3: done
  • 16:40 vgutierrez@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on lvs7001.magru.wmnet with reason: host reimage
  • 16:39 elukey@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 16:38 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2039.codfw.wmnet
  • 16:38 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti1043.eqiad.wmnet
  • 16:37 vgutierrez@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs7001.magru.wmnet (remove config only)
  • 16:37 vgutierrez@cumin1004: START - Cookbook sre.hosts.dhcp for host lvs7001.magru.wmnet (remove config only)
  • 16:36 elukey@cumin1004: START - Cookbook sre.dns.netbox
  • 16:24 slyngshede@cumin1004: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P{cp2047*} and A:cp
  • 16:24 slyngshede@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp2047.codfw.wmnet
  • 16:18 slyngshede@cumin1004: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P{cp2052*} OR P{cp2054*} and A:cp
  • 16:18 slyngshede@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp2054.codfw.wmnet
  • 16:18 vgutierrez@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs7001.magru.wmnet (push config only)
  • 16:18 vgutierrez@cumin1004: START - Cookbook sre.hosts.dhcp for host lvs7001.magru.wmnet (push config only)
  • 16:17 vgutierrez@cumin1004: START - Cookbook sre.hosts.reimage for host lvs7001.magru.wmnet with OS trixie
  • 16:13 slyngshede@cumin1004: START - Cookbook sre.cdn.roll-reboot rolling reboot on P{cp2047*} and A:cp
  • 16:10 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P{lvs7001.magru.wmnet} and A:liberica
  • 16:10 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin config_reloading P{lvs7001.magru.wmnet} and A:liberica
  • 16:09 slyngshede@cumin1004: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P{cp2046*} OR P{cp2045*} and A:cp
  • 16:09 slyngshede@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp2046.codfw.wmnet
  • 16:07 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
  • 16:06 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
  • 15:54 jiji@cumin2003: START - Cookbook sre.memcached.roll-reboot-restart rolling reboot on A:memcached-codfw
  • 15:52 jiji@cumin2003: END (PASS) - Cookbook sre.memcached.roll-reboot-restart (exit_code=0) rolling reboot on A:memcached-gutter-codfw
  • 15:40 slyngshede@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp2052.codfw.wmnet
  • 15:33 jiji@cumin2003: START - Cookbook sre.memcached.roll-reboot-restart rolling reboot on A:memcached-gutter-codfw
  • 15:31 slyngshede@cumin1004: START - Cookbook sre.cdn.roll-reboot rolling reboot on P{cp2052*} OR P{cp2054*} and A:cp
  • 15:29 slyngshede@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp2045.codfw.wmnet
  • 15:27 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
  • 15:26 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
  • 15:26 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P{ml-serve1001.eqiad.wmnet} and (A:ml-serve-master-eqiad or A:ml-serve-worker-eqiad)
  • 15:26 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1001.eqiad.wmnet
  • 15:26 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1001.eqiad.wmnet
  • 15:25 brouberol@cumin1004: END (PASS) - Cookbook sre.etcd.roll-restart-reboot (exit_code=0) rolling restart_daemons on A:ml-staging-etcd
  • 15:23 brouberol@cumin1004: START - Cookbook sre.etcd.roll-restart-reboot rolling restart_daemons on A:ml-staging-etcd
  • 15:23 brouberol@cumin1004: END (FAIL) - Cookbook sre.etcd.roll-restart-reboot (exit_code=99) rolling restart_daemons on A:ml-staging-etcd
  • 15:21 brouberol@cumin1004: START - Cookbook sre.etcd.roll-restart-reboot rolling restart_daemons on A:ml-staging-etcd
  • 15:20 herron@cumin1004: START - Cookbook sre.opensearch.roll-restart-reboot rolling reboot on A:logstash-eqiad
  • 15:20 brouberol@cumin1004: END (FAIL) - Cookbook sre.etcd.roll-restart-reboot (exit_code=99) rolling restart_daemons on A:ml-staging-etcd
  • 15:20 slyngshede@cumin1004: START - Cookbook sre.cdn.roll-reboot rolling reboot on P{cp2046*} OR P{cp2045*} and A:cp
  • 15:19 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1001.eqiad.wmnet
  • 15:17 brouberol@cumin1004: START - Cookbook sre.etcd.roll-restart-reboot rolling restart_daemons on A:ml-staging-etcd
  • 15:16 brouberol@cumin1004: END (FAIL) - Cookbook sre.etcd.roll-restart-reboot (exit_code=99) rolling restart_daemons on A:ml-staging-etcd
  • 15:16 brouberol@cumin1004: START - Cookbook sre.etcd.roll-restart-reboot rolling restart_daemons on A:ml-staging-etcd
  • 15:15 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host ml-serve1016.eqiad.wmnet (push config only)
  • 15:15 jclark@cumin1004: START - Cookbook sre.hosts.dhcp for host ml-serve1016.eqiad.wmnet (push config only)
  • 15:15 jclark@cumin1004: START - Cookbook sre.hosts.reimage for host ml-serve1016.eqiad.wmnet with OS trixie
  • 15:14 vgutierrez@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host lvs7002.magru.wmnet with OS trixie
  • 15:14 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1001.eqiad.wmnet
  • 15:14 klausman@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P{ml-serve1001.eqiad.wmnet} and (A:ml-serve-master-eqiad or A:ml-serve-worker-eqiad)
  • 15:12 jclark@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ml-serve1016.eqiad.wmnet with OS trixie
  • 15:12 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host ml-serve1016.eqiad.wmnet (remove config only)
  • 15:12 jclark@cumin1004: START - Cookbook sre.hosts.dhcp for host ml-serve1016.eqiad.wmnet (remove config only)
  • 15:12 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:ml-serve-worker-eqiad
  • 15:11 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1011.eqiad.wmnet
  • 15:11 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1011.eqiad.wmnet
  • 15:11 slyngshede@cumin1004: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P{cp2048*} OR P{cp2050*} and A:cp
  • 15:11 slyngshede@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp2050.codfw.wmnet
  • 15:10 brouberol@cumin1004: END (FAIL) - Cookbook sre.etcd.roll-restart-reboot (exit_code=99) rolling restart_daemons on A:ml-staging-etcd
  • 15:06 brouberol@cumin1004: START - Cookbook sre.etcd.roll-restart-reboot rolling restart_daemons on A:ml-staging-etcd
  • 15:05 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1011.eqiad.wmnet
  • 15:04 slyngshede@cumin1004: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P{cp2043*} OR P{cp2044*} and A:cp
  • 15:04 slyngshede@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp2044.codfw.wmnet
  • 15:00 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1011.eqiad.wmnet
  • 15:00 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1010.eqiad.wmnet
  • 15:00 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1010.eqiad.wmnet
  • 14:57 brouberol@cumin1004: END (FAIL) - Cookbook sre.etcd.roll-restart-reboot (exit_code=1) rolling restart_daemons on A:ml-staging-etcd
  • 14:57 brouberol@cumin1004: START - Cookbook sre.etcd.roll-restart-reboot rolling restart_daemons on A:ml-staging-etcd
  • 14:56 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1010.eqiad.wmnet
  • 14:54 vgutierrez@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs7002.magru.wmnet with reason: host reimage
  • 14:51 vgutierrez@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on lvs7002.magru.wmnet with reason: host reimage
  • 14:50 brouberol@cumin1004: conftool action : get/pooled; selector: name=ml-staging-etcd2001.codfw.wmnet
  • 14:48 vgutierrez@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs7002.magru.wmnet (remove config only)
  • 14:48 vgutierrez@cumin1004: START - Cookbook sre.hosts.dhcp for host lvs7002.magru.wmnet (remove config only)
  • 14:45 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1010.eqiad.wmnet
  • 14:45 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1009.eqiad.wmnet
  • 14:45 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1009.eqiad.wmnet
  • 14:37 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1009.eqiad.wmnet
  • 14:33 slyngshede@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp2048.codfw.wmnet
  • 14:32 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1009.eqiad.wmnet
  • 14:32 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1008.eqiad.wmnet
  • 14:32 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1008.eqiad.wmnet
  • 14:29 vgutierrez@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs7002.magru.wmnet (push config only)
  • 14:29 vgutierrez@cumin1004: START - Cookbook sre.hosts.dhcp for host lvs7002.magru.wmnet (push config only)
  • 14:28 vgutierrez@cumin1004: START - Cookbook sre.hosts.reimage for host lvs7002.magru.wmnet with OS trixie
  • 14:27 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1008.eqiad.wmnet
  • 14:26 urbanecm@deploy1003: Finished scap sync-world: Backport for cswiki: Add 600k celebratory logo (T442387) (duration: 16m 16s)
  • 14:25 slyngshede@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp2043.codfw.wmnet
  • 14:24 slyngshede@cumin1004: START - Cookbook sre.cdn.roll-reboot rolling reboot on P{cp2048*} OR P{cp2050*} and A:cp
  • 14:22 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1008.eqiad.wmnet
  • 14:22 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1007.eqiad.wmnet
  • 14:22 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1007.eqiad.wmnet
  • 14:21 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) depooling P{lvs7002.*} and A:liberica
  • 14:21 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin depooling P{lvs7002.*} and A:liberica
  • 14:18 urbanecm@deploy1003: urbanecm: Continuing with deployment
  • 14:17 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1007.eqiad.wmnet
  • 14:17 slyngshede@cumin1004: START - Cookbook sre.cdn.roll-reboot rolling reboot on P{cp2043*} OR P{cp2044*} and A:cp
  • 14:16 urbanecm@deploy1003: urbanecm: Backport for cswiki: Add 600k celebratory logo (T442387) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 14:14 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync
  • 14:13 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync
  • 14:12 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1007.eqiad.wmnet
  • 14:12 urbanecm@deploy1003: mwscript-k8s job started: foreachwikiindblist growthexperiments GrowthExperiments:revalidateLinkRecommendations.php --olderThan=1790175600 --verbose # T438366
  • 14:12 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1006.eqiad.wmnet
  • 14:12 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1006.eqiad.wmnet
  • 14:10 urbanecm@deploy1003: Started scap sync-world: Backport for cswiki: Add 600k celebratory logo (T442387)
  • 14:09 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
  • 14:08 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
  • 14:07 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1006.eqiad.wmnet
  • 14:06 aqu@deploy1003: Finished deploy [analytics/refinery@62e4921]: Regular analytics weekly train [analytics/refinery@62e49215] (duration: 06m 54s)
  • 14:02 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1006.eqiad.wmnet
  • 14:02 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1005.eqiad.wmnet
  • 14:02 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1005.eqiad.wmnet
  • 14:01 mlitn@deploy1003: Finished scap sync-world: Backport for Squashed diff to master (1.48.0-wmf.1, 1.47.0-wmf.24), Enable wgReaderExperimentsSemanticSearchEnabled on enwiki (T442376) (duration: 34m 09s)
  • 13:59 aqu@deploy1003: Started deploy [analytics/refinery@62e4921]: Regular analytics weekly train [analytics/refinery@62e49215]
  • 13:57 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2038.codfw.wmnet
  • 13:57 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2038.codfw.wmnet
  • 13:56 jclark@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 13:56 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1005.eqiad.wmnet
  • 13:53 jclark@cumin1004: START - Cookbook sre.dns.netbox
  • 13:53 jclark@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ml-serve1016
  • 13:53 jclark@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host ml-serve1016
  • 13:53 jclark@cumin1004: END (ERROR) - Cookbook sre.dns.netbox (exit_code=97)
  • 13:53 jclark@cumin1004: START - Cookbook sre.dns.netbox
  • 13:51 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti2038.codfw.wmnet
  • 13:51 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1005.eqiad.wmnet
  • 13:51 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1004.eqiad.wmnet
  • 13:50 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1004.eqiad.wmnet
  • 13:49 mlitn@deploy1003: mlitn: Continuing with deployment
  • 13:48 mlitn@deploy1003: mlitn: Backport for Squashed diff to master (1.48.0-wmf.1, 1.47.0-wmf.24), Enable wgReaderExperimentsSemanticSearchEnabled on enwiki (T442376) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 13:47 jclark@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 13:46 slyngshede@cumin1004: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P{cp6013*} OR P{cp6014*} OR P{cp6015*} OR P{cp6016} and A:cp
  • 13:46 slyngshede@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp6015.drmrs.wmnet
  • 13:45 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1043.eqiad.wmnet
  • 13:45 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2038.codfw.wmnet
  • 13:45 aqu@deploy1003: Finished deploy [analytics/refinery@62e4921] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@62e49215] (duration: 00m 51s)
  • 13:45 jclark@cumin1004: START - Cookbook sre.dns.netbox
  • 13:44 aqu@deploy1003: Started deploy [analytics/refinery@62e4921] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@62e49215]
  • 13:44 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1004.eqiad.wmnet
  • 13:41 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2037.codfw.wmnet
  • 13:41 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2037.codfw.wmnet
  • 13:39 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1042.eqiad.wmnet
  • 13:39 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1004.eqiad.wmnet
  • 13:39 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1003.eqiad.wmnet
  • 13:39 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1003.eqiad.wmnet
  • 13:38 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1042.eqiad.wmnet
  • 13:38 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host ml-serve1016.eqiad.wmnet (push config only)
  • 13:38 jclark@cumin1004: START - Cookbook sre.hosts.dhcp for host ml-serve1016.eqiad.wmnet (push config only)
  • 13:37 jclark@cumin1004: START - Cookbook sre.hosts.reimage for host ml-serve1016.eqiad.wmnet with OS trixie
  • 13:36 jclark@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ml-serve1016
  • 13:36 jclark@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host ml-serve1016
  • 13:36 jclark@cumin1004: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
  • 13:36 jclark@cumin1004: END (FAIL) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=99) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding ml-serve1016 to eqiad - jclark@cumin1004"
  • 13:35 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti2037.codfw.wmnet
  • 13:35 jclark@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding ml-serve1016 to eqiad - jclark@cumin1004"
  • 13:34 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1003.eqiad.wmnet
  • 13:33 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti1042.eqiad.wmnet
  • 13:33 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2037.codfw.wmnet
  • 13:31 jclark@cumin1004: START - Cookbook sre.dns.netbox
  • 13:29 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1003.eqiad.wmnet
  • 13:29 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1002.eqiad.wmnet
  • 13:29 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1002.eqiad.wmnet
  • 13:27 mlitn@deploy1003: Started scap sync-world: Backport for Squashed diff to master (1.48.0-wmf.1, 1.47.0-wmf.24), Enable wgReaderExperimentsSemanticSearchEnabled on enwiki (T442376)
  • 13:26 Amir1: dropping commons links tables from s4 sanitarium master with replication (T437278)
  • 13:24 klausman@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host ml-serve1002.eqiad.wmnet
  • 13:24 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1002.eqiad.wmnet
  • 13:24 klausman@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:ml-serve-worker-eqiad
  • 13:23 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2036.codfw.wmnet
  • 13:22 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2036.codfw.wmnet
  • 13:21 aude@deploy1003: Finished scap sync-world: Backport for Disable QuickSurvey on Special:ReadingLists (T433892) (duration: 15m 23s)
  • 13:17 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti2036.codfw.wmnet
  • 13:13 aude@deploy1003: aude: Continuing with deployment
  • 13:12 aude@deploy1003: aude: Backport for Disable QuickSurvey on Special:ReadingLists (T433892) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 13:09 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2036.codfw.wmnet
  • 13:09 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1042.eqiad.wmnet
  • 13:05 aude@deploy1003: Started scap sync-world: Backport for Disable QuickSurvey on Special:ReadingLists (T433892)
  • 13:05 slyngshede@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp6014.drmrs.wmnet
  • 12:46 krinkle@deploy1003: Finished scap sync-world: Backport for build: update required NodeJS from 24.18.0 to 26.12.1 (T442288) (1.48.0-wmf.1), Upgrade mocha to fix yargs / node 25+ incompatibility (T442317) (1.48.0-wmf.1) (duration: 33m 49s)
  • 12:34 krinkle@deploy1003: krinkle, hashar, arthurtaylor: Continuing with deployment
  • 12:33 krinkle@deploy1003: krinkle, hashar, arthurtaylor: Backport for build: update required NodeJS from 24.18.0 to 26.12.1 (T442288) (1.48.0-wmf.1), Upgrade mocha to fix yargs / node 25+ incompatibility (T442317) (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 12:30 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2035.codfw.wmnet
  • 12:30 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2035.codfw.wmnet
  • 12:27 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1041.eqiad.wmnet
  • 12:27 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1041.eqiad.wmnet
  • 12:25 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti2035.codfw.wmnet
  • 12:24 jelto@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:05:00 on phab1005.eqiad.wmnet with reason: PHP restart
  • 12:24 slyngshede@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp6013.drmrs.wmnet
  • 12:24 klausman@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-lab1002.eqiad.wmnet
  • 12:22 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti1041.eqiad.wmnet
  • 12:18 klausman@cumin1004: START - Cookbook sre.hosts.reboot-single for host ml-lab1002.eqiad.wmnet
  • 12:13 slyngshede@cumin1004: START - Cookbook sre.cdn.roll-reboot rolling reboot on P{cp6013*} OR P{cp6014*} OR P{cp6015*} OR P{cp6016} and A:cp
  • 12:13 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2035.codfw.wmnet
  • 12:12 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1041.eqiad.wmnet
  • 12:12 krinkle@deploy1003: Started scap sync-world: Backport for build: update required NodeJS from 24.18.0 to 26.12.1 (T442288) (1.48.0-wmf.1), Upgrade mocha to fix yargs / node 25+ incompatibility (T442317) (1.48.0-wmf.1)
  • 12:12 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1040.eqiad.wmnet
  • 12:12 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1040.eqiad.wmnet
  • 12:07 javiermonton@deploy1003: Failed scap deploy-service pageview-trending-relative-next: testing Flink on spiderpig, this is a staging release for test. (duration: 03m 38s)
  • 12:07 javiermonton@deploy1003: scap deploy-service pageview-trending-relative-next: [PRODUCTION/DEFAULT] dse-k8s-eqiad Rolling back pageview-trending-relative-next
  • 12:06 gkyziridis@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' .
  • 12:06 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti1040.eqiad.wmnet
  • 12:04 javiermonton@deploy1003: scap deploy-service pageview-trending-relative-next: [PRODUCTION/DEFAULT] dse-k8s-eqiad Deploying pageview-trending-relative-next
  • 12:04 javiermonton@deploy1003: Started scap deploy-service pageview-trending-relative-next: testing Flink on spiderpig, this is a staging release for test.
  • 12:02 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1040.eqiad.wmnet
  • 12:00 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
  • 12:00 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
  • 11:56 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1039.eqiad.wmnet
  • 11:56 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1039.eqiad.wmnet
  • 11:55 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply
  • 11:55 slyngshede@cumin1004: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on A:cp-upload_drmrs and A:cp
  • 11:55 slyngshede@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp6008.drmrs.wmnet
  • 11:55 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply
  • 11:55 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2050.codfw.wmnet
  • 11:55 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2050.codfw.wmnet
  • 11:52 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
  • 11:51 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
  • 11:50 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti1039.eqiad.wmnet
  • 11:49 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti2050.codfw.wmnet
  • 11:48 moritzm: installing nginx security updates
  • 11:47 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2050.codfw.wmnet
  • 11:47 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1039.eqiad.wmnet
  • 11:44 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
  • 11:42 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply
  • 11:39 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
  • 11:38 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply
  • 11:24 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply
  • 11:23 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply
  • 11:22 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply
  • 11:22 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1036.eqiad.wmnet
  • 11:22 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply
  • 11:22 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1036.eqiad.wmnet
  • 11:22 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2006.codfw.wmnet (live debug)
  • 11:20 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2049.codfw.wmnet
  • 11:19 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2049.codfw.wmnet
  • 11:18 javiermonton@deploy1003: Finished scap deploy-service pageview-trending-relative-next: testing new service in spiderpig. This is a staging release. (duration: 04m 49s)
  • 11:14 slyngshede@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp6007.drmrs.wmnet
  • 11:14 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti2049.codfw.wmnet
  • 11:14 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti1036.eqiad.wmnet
  • 11:14 javiermonton@deploy1003: scap deploy-service pageview-trending-relative-next: [PRODUCTION/DEFAULT] dse-k8s-eqiad Deploying pageview-trending-relative-next
  • 11:14 javiermonton@deploy1003: Started scap deploy-service pageview-trending-relative-next: testing new service in spiderpig. This is a staging release.
  • 11:08 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest2006.codfw.wmnet (live debug)
  • 11:07 slyngshede@cumin1004: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P{cp6009*} OR P{cp6010*} OR P{cp6011*} OR P{cp6012} and A:cp
  • 11:07 slyngshede@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp6011.drmrs.wmnet
  • 11:03 elukey@cumin1004: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['sretest2006']
  • 11:03 elukey@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['sretest2006']
  • 11:02 elukey@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2006.codfw.wmnet with OS trixie
  • 11:02 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2006.codfw.wmnet (remove config only)
  • 11:02 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest2006.codfw.wmnet (remove config only)
  • 10:58 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply
  • 10:58 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply
  • 10:56 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2006.codfw.wmnet (push config only)
  • 10:56 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest2006.codfw.wmnet (push config only)
  • 10:55 elukey@cumin1004: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS trixie
  • 10:55 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2006.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 10:46 elukey@cumin1004: START - Cookbook sre.hosts.provision for host sretest2006.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 10:39 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'llmisvc' .
  • 10:39 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
  • 10:33 slyngshede@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp6006.drmrs.wmnet
  • 10:31 kevinbazira@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
  • 10:31 kevinbazira@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
  • 10:26 slyngshede@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp6010.drmrs.wmnet
  • 09:56 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen: apply
  • 09:55 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen: apply
  • 09:55 slyngshede@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp6005.drmrs.wmnet
  • 09:53 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply
  • 09:52 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply
  • 09:51 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'llmisvc' .
  • 09:51 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
  • 09:50 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
  • 09:48 arthurtaylor@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikidata-query-gui: apply
  • 09:48 arthurtaylor@deploy1003: helmfile [codfw] START helmfile.d/services/wikidata-query-gui: apply
  • 09:48 arthurtaylor@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply
  • 09:47 arthurtaylor@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply
  • 09:47 arthurtaylor@deploy1003: helmfile [staging] DONE helmfile.d/services/wikidata-query-gui: apply
  • 09:47 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1036.eqiad.wmnet
  • 09:47 arthurtaylor@deploy1003: helmfile [staging] START helmfile.d/services/wikidata-query-gui: apply
  • 09:45 slyngshede@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp6009.drmrs.wmnet
  • 09:43 arthurtaylor@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikidata-query-gui: apply
  • 09:43 arthurtaylor@deploy1003: helmfile [codfw] START helmfile.d/services/wikidata-query-gui: apply
  • 09:43 arthurtaylor@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply
  • 09:43 arthurtaylor@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply
  • 09:42 arthurtaylor@deploy1003: helmfile [staging] DONE helmfile.d/services/wikidata-query-gui: apply
  • 09:41 arthurtaylor@deploy1003: helmfile [staging] START helmfile.d/services/wikidata-query-gui: apply
  • 09:40 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2049.codfw.wmnet
  • 09:36 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2048.codfw.wmnet
  • 09:34 slyngshede@cumin1004: START - Cookbook sre.cdn.roll-reboot rolling reboot on P{cp6009*} OR P{cp6010*} OR P{cp6011*} OR P{cp6012} and A:cp
  • 09:32 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1057.eqiad.wmnet
  • 09:32 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1057.eqiad.wmnet
  • 09:32 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2048.codfw.wmnet
  • 09:31 klausman@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host ml-serve1016.eqiad.wmnet with OS trixie
  • 09:31 klausman@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host ml-serve1016.eqiad.wmnet (remove config only)
  • 09:31 klausman@cumin1004: START - Cookbook sre.hosts.dhcp for host ml-serve1016.eqiad.wmnet (remove config only)
  • 09:30 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
  • 09:28 slyngshede@cumin1004: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P{cp6001*} OR P{cp6002*} and A:cp
  • 09:28 slyngshede@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp6002.drmrs.wmnet
  • 09:28 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'llmisvc' .
  • 09:28 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
  • 09:28 dkertesz@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-text_eqiad and A:cp - upgrade haproxy to 3.2.26-1 (T441435)
  • 09:27 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti2048.codfw.wmnet
  • 09:27 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti1057.eqiad.wmnet
  • 09:26 kevinbazira@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
  • 09:25 kevinbazira@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
  • 09:22 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1057.eqiad.wmnet
  • 09:21 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2048.codfw.wmnet
  • 09:17 mvernon@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1100.eqiad.wmnet
  • 09:16 moritzm: installing brotli security updates
  • 09:16 klausman@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host ml-serve1016.eqiad.wmnet (push config only)
  • 09:16 klausman@cumin1004: START - Cookbook sre.hosts.dhcp for host ml-serve1016.eqiad.wmnet (push config only)
  • 09:15 klausman@cumin1004: START - Cookbook sre.hosts.reimage for host ml-serve1016.eqiad.wmnet with OS trixie
  • 09:15 slyngshede@cumin1004: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P{cp5025*} OR P{cp5026*} and A:cp
  • 09:15 slyngshede@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp5026.eqsin.wmnet
  • 09:14 slyngshede@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp6004.drmrs.wmnet
  • 09:10 mvernon@cumin1004: START - Cookbook sre.hosts.reboot-single for host ms-be1100.eqiad.wmnet
  • 09:10 mvernon@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1099.eqiad.wmnet
  • 09:09 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1056.eqiad.wmnet
  • 09:08 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1056.eqiad.wmnet
  • 09:04 dkertesz@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-upload_eqiad and A:cp - upgrade haproxy to 3.2.26-1 (T441435)
  • 09:03 mvernon@cumin1004: START - Cookbook sre.hosts.reboot-single for host ms-be1099.eqiad.wmnet
  • 09:03 mvernon@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1098.eqiad.wmnet
  • 09:03 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2032.codfw.wmnet
  • 09:03 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti1056.eqiad.wmnet
  • 09:03 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2032.codfw.wmnet
  • 08:58 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2098.codfw.wmnet
  • 08:57 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1056.eqiad.wmnet
  • 08:57 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti2032.codfw.wmnet
  • 08:56 mvernon@cumin1004: START - Cookbook sre.hosts.reboot-single for host ms-be1098.eqiad.wmnet
  • 08:56 mvernon@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1097.eqiad.wmnet
  • 08:56 elukey@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2006.codfw.wmnet with OS trixie
  • 08:56 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2006.codfw.wmnet (remove config only)
  • 08:56 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest2006.codfw.wmnet (remove config only)
  • 08:52 mvernon@cumin2003: START - Cookbook sre.hosts.reboot-single for host ms-be2098.codfw.wmnet
  • 08:52 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2097.codfw.wmnet
  • 08:50 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2006.codfw.wmnet (push config only)
  • 08:50 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest2006.codfw.wmnet (push config only)
  • 08:50 slyngshede@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp6001.drmrs.wmnet
  • 08:50 elukey@cumin1004: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS trixie
  • 08:49 elukey@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2006.codfw.wmnet with OS trixie
  • 08:49 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2006.codfw.wmnet (remove config only)
  • 08:49 mvernon@cumin1004: START - Cookbook sre.hosts.reboot-single for host ms-be1097.eqiad.wmnet
  • 08:49 mvernon@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1096.eqiad.wmnet
  • 08:49 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest2006.codfw.wmnet (remove config only)
  • 08:45 mvernon@cumin2003: START - Cookbook sre.hosts.reboot-single for host ms-be2097.codfw.wmnet
  • 08:45 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2096.codfw.wmnet
  • 08:45 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2006.codfw.wmnet (push config only)
  • 08:45 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest2006.codfw.wmnet (push config only)
  • 08:45 elukey@cumin1004: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS trixie
  • 08:42 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1055.eqiad.wmnet
  • 08:42 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1055.eqiad.wmnet
  • 08:42 mvernon@cumin1004: START - Cookbook sre.hosts.reboot-single for host ms-be1096.eqiad.wmnet
  • 08:42 mvernon@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1095.eqiad.wmnet
  • 08:39 slyngshede@cumin1004: START - Cookbook sre.cdn.roll-reboot rolling reboot on P{cp6001*} OR P{cp6002*} and A:cp
  • 08:39 mvernon@cumin2003: START - Cookbook sre.hosts.reboot-single for host ms-be2096.codfw.wmnet
  • 08:39 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2095.codfw.wmnet
  • 08:36 dkertesz@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_eqiad and A:cp - upgrade haproxy to 3.2.26-1 (T441435)
  • 08:36 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2032.codfw.wmnet
  • 08:36 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti1055.eqiad.wmnet
  • 08:36 dkertesz@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_eqiad and A:cp - upgrade haproxy to 3.2.26-1 (T441435)
  • 08:35 mvernon@cumin1004: START - Cookbook sre.hosts.reboot-single for host ms-be1095.eqiad.wmnet
  • 08:35 mvernon@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1094.eqiad.wmnet
  • 08:35 slyngshede@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp6003.drmrs.wmnet
  • 08:34 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1055.eqiad.wmnet
  • 08:33 slyngshede@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp5025.eqsin.wmnet
  • 08:32 mvernon@cumin2003: START - Cookbook sre.hosts.reboot-single for host ms-be2095.codfw.wmnet
  • 08:32 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2094.codfw.wmnet
  • 08:30 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2031.codfw.wmnet
  • 08:28 mvernon@cumin1004: START - Cookbook sre.hosts.reboot-single for host ms-be1094.eqiad.wmnet
  • 08:28 mvernon@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1093.eqiad.wmnet
  • 08:27 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2031.codfw.wmnet
  • 08:26 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1054.eqiad.wmnet
  • 08:26 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1054.eqiad.wmnet
  • 08:26 mvernon@cumin2003: START - Cookbook sre.hosts.reboot-single for host ms-be2094.codfw.wmnet
  • 08:26 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2093.codfw.wmnet
  • 08:23 slyngshede@cumin1004: START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-upload_drmrs and A:cp
  • 08:22 slyngshede@cumin1004: START - Cookbook sre.cdn.roll-reboot rolling reboot on P{cp5025*} OR P{cp5026*} and A:cp
  • 08:21 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti1054.eqiad.wmnet
  • 08:21 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti2031.codfw.wmnet
  • 08:19 mvernon@cumin2003: START - Cookbook sre.hosts.reboot-single for host ms-be2093.codfw.wmnet
  • 08:19 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2092.codfw.wmnet
  • 08:19 mvernon@cumin1004: START - Cookbook sre.hosts.reboot-single for host ms-be1093.eqiad.wmnet
  • 08:18 mvernon@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1092.eqiad.wmnet
  • 08:16 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2031.codfw.wmnet
  • 08:14 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1054.eqiad.wmnet
  • 08:12 mvernon@cumin2003: START - Cookbook sre.hosts.reboot-single for host ms-be2092.codfw.wmnet
  • 08:11 mvernon@cumin1004: START - Cookbook sre.hosts.reboot-single for host ms-be1092.eqiad.wmnet
  • 08:05 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1053.eqiad.wmnet
  • 08:04 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1053.eqiad.wmnet
  • 08:01 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2045.codfw.wmnet
  • 08:01 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2045.codfw.wmnet
  • 07:59 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti1053.eqiad.wmnet
  • 07:55 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti2045.codfw.wmnet
  • 07:55 jmm@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on logstash2023.codfw.wmnet with reason: reboot
  • 07:54 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1053.eqiad.wmnet
  • 07:53 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2045.codfw.wmnet
  • 07:48 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
  • 07:48 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
  • 07:44 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2046.codfw.wmnet
  • 07:44 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2046.codfw.wmnet
  • 07:43 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1035.eqiad.wmnet
  • 07:42 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1035.eqiad.wmnet
  • 07:38 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti2046.codfw.wmnet
  • 07:35 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2046.codfw.wmnet
  • 07:35 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti1035.eqiad.wmnet
  • 07:35 jmm@cumin2003: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti2045.codfw.wmnet
  • 07:33 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2045.codfw.wmnet
  • 07:33 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1035.eqiad.wmnet
  • 07:16 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1039: Back in prod
  • 06:50 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'llmisvc' .
  • 06:50 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
  • 06:45 marostegui@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section x3
  • 06:44 kevinbazira@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
  • 06:44 kevinbazira@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
  • 06:44 marostegui@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section x3
  • 06:38 marostegui@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section x1
  • 06:37 marostegui@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section x1
  • 06:31 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool es1039: Back in prod
  • 06:31 marostegui@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool es1039: Back in prod
  • 06:31 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool es1039: Back in prod
  • 00:00 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
  • 00:00 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
  • 00:00 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
  • 00:00 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: sync

2026-10-25

  • 18:24 urbanecm: [urbanecm@deploy1003 ~]$ mwscript-k8s -f -- extensions/CentralAuth/maintenance/attachAccount.php --wiki=banwiki --username='DraconicDark' # T442325
  • 12:00 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
  • 12:00 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
  • 12:00 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
  • 12:00 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: sync
  • 00:00 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
  • 00:00 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: sync

2026-10-24

  • 12:00 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
  • 12:00 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
  • 02:52 sbassett: Deployed security fix for T442250 to wmf.24
  • 02:37 sbassett: Deployed security fix for T442250 to wmf.1
  • 00:57 bd808: tools.stashbot Killed everything in k8s namespace; restarted webservice with php8.4 runtime; restarted bot with `./bin/stashbot.sh start`
  • 00:46 bd808: tools.quarry Rebuild with Tool:Containers#Redirect_container
  • 00:40 bd808: tools.pygments Webservice restart to reconnect with front proxy
  • 00:37 bd808: tools.toolhub-gadget-demo Webservice restart to reconnect with front proxy
  • 00:29 bd808: tools.toolinfo-scraper Restart to pick up missing envvars
  • 00:28 bd808: tools.toolhub Restart with php8.4 runtime
  • 00:24 bd808: tools.pv Replaced neat lighttpd hacks with tool-containers/rproxy:latest
  • 00:00 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
  • 00:00 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
  • 00:00 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
  • 00:00 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: sync

2026-10-23

  • 23:45 bd808: tools.prototyper-proxy Restart to switch to php8.4 runtime
  • 23:43 bd808: tools.phabulous Restart to switch to php8.4 runtime
  • 23:40 bd808: tools.phabricator-bug-status Webservice restart to reconnect with front proxy
  • 23:32 bd808: tools.nagf Restarted with php8.4 runtime; dirty diff changes left in ~/src/nagf
  • 23:22 bd808: tools.mysql-php-session-test Restart with php8.4 runtime
  • 23:17 bd808: tools.my-first-flask-tool Restarted with python3.13 runtime
  • 23:12 bd808: tools.my-first-flask-oauth-tool Webservice restart to reconnect with front proxy
  • 22:43 bd808: tools.mwdemo update.php not working because MediaWiki apparently requires a newer MariaDB than ToolsDB provides. :/
  • 22:42 bd808: tools.mwdemo Fixed broken git checkout for VE extension
  • 22:16 dancy@deploy1003: Installation of scap version "4.305.0" completed for 3 hosts
  • 22:14 dancy@deploy1003: Installing scap version "4.305.0" for 3 host(s)
  • 21:27 mutante: etherpad is currently shut down (T442232)
  • 20:31 dzahn@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4 days, 0:00:00 on etherpad2002.codfw.wmnet,etherpad1004.eqiad.wmnet with reason: temp disabled
  • 20:12 btullis@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4 days, 0:00:00 on etherpad2002.codfw.wmnet with reason: T442177 shutting down etherpad
  • 20:11 btullis@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4 days, 0:00:00 on etherpad1004.eqiad.wmnet with reason: T442177 shutting down etherpad
  • 18:32 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/eventstreams-internal: apply
  • 18:31 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/eventstreams-internal: apply
  • 18:29 btullis@deploy1003: helmfile [eqiad] DONE helmfile.d/services/eventstreams: apply
  • 18:28 btullis@deploy1003: helmfile [eqiad] START helmfile.d/services/eventstreams: apply
  • 18:27 btullis@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventstreams: apply
  • 18:26 btullis@deploy1003: helmfile [codfw] START helmfile.d/services/eventstreams: apply
  • 18:25 btullis@deploy1003: helmfile [staging] DONE helmfile.d/services/eventstreams: apply
  • 18:24 btullis@deploy1003: helmfile [staging] START helmfile.d/services/eventstreams: apply
  • 17:03 mutante: contint1002 - purging PHP packages - restarting httpd
  • 17:00 mutante: contint2002 - purging PHP packages
  • 16:02 bking@cumin2003: END (PASS) - Cookbook sre.elasticsearch.restart-nginx (exit_code=0) rolling restart_daemons on A:cloudelastic
  • 16:00 bking@cumin2003: START - Cookbook sre.elasticsearch.restart-nginx rolling restart_daemons on A:cloudelastic
  • 16:00 bking@cumin2003: END (PASS) - Cookbook sre.elasticsearch.restart-nginx (exit_code=0) rolling restart_daemons on A:relforge
  • 15:59 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply
  • 15:59 bking@cumin2003: START - Cookbook sre.elasticsearch.restart-nginx rolling restart_daemons on A:relforge
  • 15:59 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply
  • 15:58 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthbook: apply
  • 15:58 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook: apply
  • 15:48 dancy@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: Deployments locked for the week (duration: 16m 28s)
  • 15:48 dancy@deploy1003: Forcefully removed global lock: Deployments locked for the week
  • 15:39 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-wdqs2005.codfw.wmnet with OS bookworm
  • 15:36 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-wdqs1004.eqiad.wmnet with OS bookworm
  • 15:31 dancy@deploy1003: Locking from deployment [ALL REPOSITORIES]: Deployments locked for the week
  • 15:31 bking@cumin2003: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: apply kernel updates - bking@cumin2003 - T441838
  • 15:30 dancy@deploy1003: Finished scap sync-world: Backport for Upgrading composer/ca-bundle (1.5.14 => 1.6.0) (1.48.0-wmf.1) (duration: 10m 33s)
  • 15:26 oblivian@deploy1003: helmfile [eqiad] DONE helmfile.d/services/citoid: apply
  • 15:26 oblivian@deploy1003: helmfile [eqiad] START helmfile.d/services/citoid: apply
  • 15:26 oblivian@deploy1003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply
  • 15:25 oblivian@deploy1003: helmfile [eqiad] START helmfile.d/services/zotero: apply
  • 15:25 dancy@deploy1003: dancy, zabe: Continuing with deployment
  • 15:24 dancy@deploy1003: dancy, zabe: Backport for Upgrading composer/ca-bundle (1.5.14 => 1.6.0) (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 15:24 oblivian@deploy1003: helmfile [codfw] DONE helmfile.d/services/citoid: apply
  • 15:23 oblivian@deploy1003: helmfile [codfw] START helmfile.d/services/citoid: apply
  • 15:23 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-wdqs2005.codfw.wmnet with reason: host reimage
  • 15:23 oblivian@deploy1003: helmfile [staging] DONE helmfile.d/services/citoid: apply
  • 15:22 oblivian@deploy1003: helmfile [staging] START helmfile.d/services/citoid: apply
  • 15:22 oblivian@deploy1003: helmfile [staging] DONE helmfile.d/services/zotero: apply
  • 15:22 oblivian@deploy1003: helmfile [staging] START helmfile.d/services/zotero: apply
  • 15:21 oblivian@deploy1003: helmfile [codfw] DONE helmfile.d/services/zotero: apply
  • 15:21 oblivian@deploy1003: helmfile [codfw] START helmfile.d/services/zotero: apply
  • 15:20 dancy@deploy1003: Started scap sync-world: Backport for Upgrading composer/ca-bundle (1.5.14 => 1.6.0) (1.48.0-wmf.1)
  • 15:19 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-wdqs1004.eqiad.wmnet with reason: host reimage
  • 15:19 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/zotero: apply
  • 15:19 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/zotero: apply
  • 15:17 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/citoid: apply
  • 15:17 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/citoid: apply
  • 15:15 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-wdqs2005.codfw.wmnet with reason: host reimage
  • 15:15 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-wdqs1004.eqiad.wmnet with reason: host reimage
  • 15:10 dancy@deploy1003: Installation of scap version "4.304.0" completed for 3 hosts
  • 15:08 dancy@deploy1003: Installing scap version "4.304.0" for 3 host(s)
  • 15:08 dancy@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: Deployments locked for the week (duration: 1084m 20s)
  • 15:07 dancy@deploy1003: Forcefully removing global lock: Deployments locked for the week
  • 15:07 bking@cumin2003: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: apply kernel updates - bking@cumin2003 - T441838
  • 15:01 bking@cumin2003: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: apply security manager updates - bking@cumin2003 - T324335
  • 14:59 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-wdqs2005.codfw.wmnet with OS bookworm
  • 14:58 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-wdqs1004.eqiad.wmnet with OS bookworm
  • 14:58 btullis@cumin1004: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['dse-k8s-wdqs1004.eqiad.wmnet']
  • 14:54 bking@cumin2003: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: apply security manager updates - bking@cumin2003 - T324335
  • 14:50 btullis@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['dse-k8s-wdqs1004.eqiad.wmnet']
  • 14:47 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host dse-k8s-wdqs2005.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 14:46 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host dse-k8s-wdqs1004.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 14:38 btullis@cumin1004: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2005.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 14:38 btullis@cumin1004: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs1004.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 13:57 btullis@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on an-presto1006.eqiad.wmnet with reason: Recreate data RAID without failed disk
  • 13:44 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P{dse-k8s-wdqs*} and (A:dse-k8s-master-codfw or A:dse-k8s-worker-codfw)
  • 13:44 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs-test2001.codfw.wmnet
  • 13:44 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs-test2001.codfw.wmnet
  • 13:20 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P{dse-k8s-worker2*} and (A:dse-k8s-master-codfw or A:dse-k8s-worker-codfw)
  • 13:20 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2011.codfw.wmnet
  • 13:20 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2011.codfw.wmnet
  • 13:14 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2011.codfw.wmnet
  • 13:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2011.codfw.wmnet
  • 13:09 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2010.codfw.wmnet
  • 13:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2010.codfw.wmnet
  • 13:03 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2010.codfw.wmnet
  • 13:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2010.codfw.wmnet
  • 13:03 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2009.codfw.wmnet
  • 13:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2009.codfw.wmnet
  • 12:57 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2009.codfw.wmnet
  • 12:56 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2009.codfw.wmnet
  • 12:56 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2008.codfw.wmnet
  • 12:56 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2008.codfw.wmnet
  • 12:51 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2008.codfw.wmnet
  • 12:50 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2008.codfw.wmnet
  • 12:50 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2007.codfw.wmnet
  • 12:50 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2007.codfw.wmnet
  • 12:45 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2007.codfw.wmnet
  • 12:44 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2007.codfw.wmnet
  • 12:44 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2006.codfw.wmnet
  • 12:44 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2006.codfw.wmnet
  • 12:39 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2006.codfw.wmnet
  • 12:38 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2006.codfw.wmnet
  • 12:38 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2005.codfw.wmnet
  • 12:38 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2005.codfw.wmnet
  • 12:33 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2005.codfw.wmnet
  • 12:32 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2005.codfw.wmnet
  • 12:32 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2004.codfw.wmnet
  • 12:32 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2004.codfw.wmnet
  • 12:30 cwilliams@cumin1004: END (PASS) - Cookbook sre.mysql.test-cluster (exit_code=0)
  • 12:30 cwilliams@cumin1004: START - Cookbook sre.mysql.test-cluster
  • 12:29 cwilliams@cumin1004: END (PASS) - Cookbook sre.mysql.test-cluster (exit_code=0)
  • 12:29 cwilliams@cumin1004: START - Cookbook sre.mysql.test-cluster
  • 12:27 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2004.codfw.wmnet
  • 12:26 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2004.codfw.wmnet
  • 12:26 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2003.codfw.wmnet
  • 12:26 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2003.codfw.wmnet
  • 12:25 cwilliams@cumin1004: END (PASS) - Cookbook sre.mysql.test-cluster (exit_code=0)
  • 12:25 cwilliams@cumin1004: START - Cookbook sre.mysql.test-cluster
  • 12:24 cwilliams@cumin1004: END (FAIL) - Cookbook sre.mysql.test-cluster (exit_code=99)
  • 12:24 cwilliams@cumin1004: START - Cookbook sre.mysql.test-cluster
  • 12:21 cwilliams@cumin1004: END (PASS) - Cookbook sre.mysql.test-cluster (exit_code=0)
  • 12:21 cwilliams@cumin1004: START - Cookbook sre.mysql.test-cluster
  • 12:21 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2003.codfw.wmnet
  • 12:21 cwilliams@cumin1004: END (PASS) - Cookbook sre.mysql.test-cluster (exit_code=0)
  • 12:21 cwilliams@cumin1004: START - Cookbook sre.mysql.test-cluster
  • 12:21 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 12:17 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 12:17 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 12:16 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2003.codfw.wmnet
  • 12:16 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2002.codfw.wmnet
  • 12:16 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2002.codfw.wmnet
  • 12:14 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 12:09 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2002.codfw.wmnet
  • 12:04 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2002.codfw.wmnet
  • 12:04 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2001.codfw.wmnet
  • 12:04 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2001.codfw.wmnet
  • 12:00 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
  • 12:00 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
  • 12:00 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
  • 12:00 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: sync
  • 11:56 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2001.codfw.wmnet
  • 11:54 ayounsi@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
  • 11:53 ayounsi@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
  • 11:52 ayounsi@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary
  • 11:51 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2001.codfw.wmnet
  • 11:51 ayounsi@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
  • 11:51 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P{dse-k8s-worker2*} and (A:dse-k8s-master-codfw or A:dse-k8s-worker-codfw)
  • 11:49 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from wcqs2003 to dse-k8s-wdqs2005
  • 11:49 btullis@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-wdqs2005
  • 11:31 btullis@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-wdqs2005
  • 11:31 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-wdqs2005 on all recursors
  • 11:31 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-wdqs2005 on all recursors
  • 11:31 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 11:31 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming wcqs2003 to dse-k8s-wdqs2005 - btullis@cumin1004"
  • 11:29 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming wcqs2003 to dse-k8s-wdqs2005 - btullis@cumin1004"
  • 11:19 btullis@cumin1004: START - Cookbook sre.dns.netbox
  • 11:18 btullis@cumin1004: START - Cookbook sre.hosts.rename from wcqs2003 to dse-k8s-wdqs2005
  • 11:01 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from wcqs1003 to dse-k8s-wdqs1004
  • 11:01 btullis@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-wdqs1004
  • 10:58 btullis@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-wdqs1004
  • 10:58 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-wdqs1004 on all recursors
  • 10:58 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-wdqs1004 on all recursors
  • 10:58 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 10:58 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming wcqs1003 to dse-k8s-wdqs1004 - btullis@cumin1004"
  • 10:57 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming wcqs1003 to dse-k8s-wdqs1004 - btullis@cumin1004"
  • 10:53 btullis@cumin1004: START - Cookbook sre.dns.netbox
  • 10:53 btullis@cumin1004: START - Cookbook sre.hosts.rename from wcqs1003 to dse-k8s-wdqs1004
  • 10:45 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs-test2001.codfw.wmnet
  • 10:44 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs-test2001.codfw.wmnet
  • 10:44 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs2004.codfw.wmnet
  • 10:44 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs2004.codfw.wmnet
  • 10:39 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs2004.codfw.wmnet
  • 10:38 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs2004.codfw.wmnet
  • 10:38 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs2003.codfw.wmnet
  • 10:38 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs2003.codfw.wmnet
  • 10:33 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs2003.codfw.wmnet
  • 10:32 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs2003.codfw.wmnet
  • 10:32 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs2002.codfw.wmnet
  • 10:32 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs2002.codfw.wmnet
  • 10:27 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs2002.codfw.wmnet
  • 10:26 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs2002.codfw.wmnet
  • 10:26 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs2001.codfw.wmnet
  • 10:26 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs2001.codfw.wmnet
  • 10:21 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs2001.codfw.wmnet
  • 10:20 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs2001.codfw.wmnet
  • 10:20 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P{dse-k8s-wdqs*} and (A:dse-k8s-master-codfw or A:dse-k8s-worker-codfw)
  • 10:20 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P{dse-k8s-wdqs-test1001*} and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad)
  • 10:20 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs-test1001.eqiad.wmnet
  • 10:20 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs-test1001.eqiad.wmnet
  • 10:13 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs-test1001.eqiad.wmnet
  • 10:13 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs-test1001.eqiad.wmnet
  • 10:13 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P{dse-k8s-wdqs-test1001*} and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad)
  • 10:12 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P{dse-k8s-wdqs100[1-3]*} and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad)
  • 10:12 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1003.eqiad.wmnet
  • 10:12 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1003.eqiad.wmnet
  • 10:06 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1003.eqiad.wmnet
  • 10:06 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1003.eqiad.wmnet
  • 10:06 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1002.eqiad.wmnet
  • 10:06 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1002.eqiad.wmnet
  • 10:05 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host search-loader2002.codfw.wmnet
  • 10:01 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host search-loader2002.codfw.wmnet
  • 10:01 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host search-loader1002.eqiad.wmnet
  • 10:00 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1002.eqiad.wmnet
  • 10:00 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1002.eqiad.wmnet
  • 09:59 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1001.eqiad.wmnet
  • 09:59 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1001.eqiad.wmnet
  • 09:57 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host search-loader1002.eqiad.wmnet
  • 09:54 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1001.eqiad.wmnet
  • 09:54 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1001.eqiad.wmnet
  • 09:53 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P{dse-k8s-wdqs100[1-3]*} and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad)
  • 09:49 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P{dse-k8s-worker100[1-4]*} and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad)
  • 09:49 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1004.eqiad.wmnet
  • 09:49 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1004.eqiad.wmnet
  • 09:43 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1004.eqiad.wmnet
  • 09:38 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1004.eqiad.wmnet
  • 09:38 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1003.eqiad.wmnet
  • 09:38 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1003.eqiad.wmnet
  • 09:32 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1003.eqiad.wmnet
  • 09:08 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs1032.eqiad.wmnet
  • 09:02 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host wdqs1032.eqiad.wmnet
  • 09:02 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs1031.eqiad.wmnet
  • 09:01 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1003.eqiad.wmnet
  • 09:01 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1002.eqiad.wmnet
  • 09:01 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1002.eqiad.wmnet
  • 08:55 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1002.eqiad.wmnet
  • 08:55 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host wdqs1031.eqiad.wmnet
  • 08:55 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs1030.eqiad.wmnet
  • 08:54 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1002.eqiad.wmnet
  • 08:54 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1001.eqiad.wmnet
  • 08:54 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1001.eqiad.wmnet
  • 08:48 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1001.eqiad.wmnet
  • 08:47 marostegui@cumin1004: dbctl commit (dc=all): 'Change werights for db2243', diff saved to https://phabricator.wikimedia.org/P96985 and previous config saved to /var/cache/conftool/dbconfig/20261023-084725-marostegui.json
  • 08:47 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host wdqs1030.eqiad.wmnet
  • 08:47 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs1029.eqiad.wmnet
  • 08:47 marostegui@cumin1004: dbctl commit (dc=all): 'Change werights for db1257 and db1260', diff saved to https://phabricator.wikimedia.org/P96984 and previous config saved to /var/cache/conftool/dbconfig/20261023-084659-marostegui.json
  • 08:40 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host wdqs1029.eqiad.wmnet
  • 08:34 marostegui@cumin1004: dbctl commit (dc=all): 'Depool es1039', diff saved to https://phabricator.wikimedia.org/P96983 and previous config saved to /var/cache/conftool/dbconfig/20261023-083409-marostegui.json
  • 08:28 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host krb1004.eqiad.wmnet
  • 08:24 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host krb1004.eqiad.wmnet
  • 08:18 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1001.eqiad.wmnet
  • 08:18 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P{dse-k8s-worker100[1-4]*} and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad)
  • 08:11 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on dbproxy1029.eqiad.wmnet with reason: Restart
  • 08:06 elukey@cumin1004: END (PASS) - Cookbook sre.pki.restart-reboot (exit_code=0) rolling reboot on A:pki
  • 07:41 elukey@cumin1004: START - Cookbook sre.pki.restart-reboot rolling reboot on A:pki
  • 04:18 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:dse-k8s-worker-eqiad
  • 04:18 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1044.eqiad.wmnet
  • 04:18 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1044.eqiad.wmnet
  • 04:10 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1044.eqiad.wmnet
  • 03:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1044.eqiad.wmnet
  • 03:40 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1043.eqiad.wmnet
  • 03:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1043.eqiad.wmnet
  • 03:34 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1043.eqiad.wmnet
  • 03:04 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1043.eqiad.wmnet
  • 03:04 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1042.eqiad.wmnet
  • 03:04 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1042.eqiad.wmnet
  • 02:57 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1042.eqiad.wmnet
  • 02:55 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1042.eqiad.wmnet
  • 02:55 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1041.eqiad.wmnet
  • 02:54 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1041.eqiad.wmnet
  • 02:51 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1041.eqiad.wmnet
  • 02:51 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1041.eqiad.wmnet
  • 02:51 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1040.eqiad.wmnet
  • 02:51 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1040.eqiad.wmnet
  • 02:45 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1040.eqiad.wmnet
  • 02:15 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1040.eqiad.wmnet
  • 02:15 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1039.eqiad.wmnet
  • 02:15 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1039.eqiad.wmnet
  • 01:58 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1039.eqiad.wmnet
  • 01:56 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1039.eqiad.wmnet
  • 01:56 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1028.eqiad.wmnet
  • 01:56 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1028.eqiad.wmnet
  • 01:50 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1028.eqiad.wmnet
  • 01:49 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1028.eqiad.wmnet
  • 01:49 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1027.eqiad.wmnet
  • 01:49 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1027.eqiad.wmnet
  • 01:42 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1027.eqiad.wmnet
  • 01:12 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1027.eqiad.wmnet
  • 01:12 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1026.eqiad.wmnet
  • 01:12 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1026.eqiad.wmnet
  • 01:08 mutante: contint1002/2002: reverted manual httpd config hack and graceful'ed apache - like https://gerrit.wikimedia.org/r/c/operations/puppet/+/1357917
  • 01:05 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1026.eqiad.wmnet
  • 01:05 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1026.eqiad.wmnet
  • 01:05 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1025.eqiad.wmnet
  • 01:05 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1025.eqiad.wmnet
  • 00:58 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1025.eqiad.wmnet
  • 00:28 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1025.eqiad.wmnet
  • 00:28 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1024.eqiad.wmnet
  • 00:28 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1024.eqiad.wmnet
  • 00:21 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1024.eqiad.wmnet
  • 00:20 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1024.eqiad.wmnet
  • 00:20 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1023.eqiad.wmnet
  • 00:20 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1023.eqiad.wmnet
  • 00:13 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1023.eqiad.wmnet
  • 00:12 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1023.eqiad.wmnet
  • 00:12 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1022.eqiad.wmnet
  • 00:12 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1022.eqiad.wmnet
  • 00:06 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1022.eqiad.wmnet
  • 00:00 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
  • 00:00 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: sync

2026-10-22

  • 23:36 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1022.eqiad.wmnet
  • 23:36 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1021.eqiad.wmnet
  • 23:36 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1021.eqiad.wmnet
  • 23:29 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1021.eqiad.wmnet
  • 22:59 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1021.eqiad.wmnet
  • 22:59 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1020.eqiad.wmnet
  • 22:59 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1020.eqiad.wmnet
  • 22:41 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1020.eqiad.wmnet
  • 22:39 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1020.eqiad.wmnet
  • 22:39 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1019.eqiad.wmnet
  • 22:39 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1019.eqiad.wmnet
  • 22:32 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1019.eqiad.wmnet
  • 22:32 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1019.eqiad.wmnet
  • 22:32 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1018.eqiad.wmnet
  • 22:32 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1018.eqiad.wmnet
  • 22:26 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1018.eqiad.wmnet
  • 22:25 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1018.eqiad.wmnet
  • 22:25 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1017.eqiad.wmnet
  • 22:25 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1017.eqiad.wmnet
  • 22:14 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1017.eqiad.wmnet
  • 22:13 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1017.eqiad.wmnet
  • 22:13 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1016.eqiad.wmnet
  • 22:13 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1016.eqiad.wmnet
  • 22:07 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1016.eqiad.wmnet
  • 21:37 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1016.eqiad.wmnet
  • 21:37 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1015.eqiad.wmnet
  • 21:37 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1015.eqiad.wmnet
  • 21:31 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1015.eqiad.wmnet
  • 21:24 akhatun@deploy1003: helmfile [staging] DONE helmfile.d/services/lac-lambdas: sync
  • 21:24 akhatun@deploy1003: helmfile [staging] START helmfile.d/services/lac-lambdas: sync
  • 21:03 dancy@deploy1003: Locking from deployment [ALL REPOSITORIES]: Deployments locked for the week
  • 21:03 dancy@deploy1003: Finished scap sync-world: Backport for wikisource-ocr-langs: Lower timeout, cache absence, set lockTSE+busyValue (T442064) (1.48.0-wmf.1) (duration: 08m 23s)
  • 21:01 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1015.eqiad.wmnet
  • 21:01 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1014.eqiad.wmnet
  • 21:01 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1014.eqiad.wmnet
  • 20:57 dancy@deploy1003: dancy: Continuing with deployment
  • 20:57 dancy@deploy1003: dancy: Backport for wikisource-ocr-langs: Lower timeout, cache absence, set lockTSE+busyValue (T442064) (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 20:55 dancy@deploy1003: Started scap sync-world: Backport for wikisource-ocr-langs: Lower timeout, cache absence, set lockTSE+busyValue (T442064) (1.48.0-wmf.1)
  • 20:53 dancy@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: Deployments locked for the week (duration: 11m 08s)
  • 20:53 dancy@deploy1003: Forcefully removing global lock: Deployments locked for the week
  • 20:45 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1014.eqiad.wmnet
  • 20:42 dancy@deploy1003: Locking from deployment [ALL REPOSITORIES]: Deployments locked for the week
  • 20:42 dancy@deploy1003: Finished scap sync-world: Backport for wikisource-ocr-langs: Lower timeout, cache absence, set lockTSE+busyValue (T442064) (1.47.0-wmf.24) (duration: 08m 51s)
  • 20:41 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1014.eqiad.wmnet
  • 20:40 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1013.eqiad.wmnet
  • 20:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1013.eqiad.wmnet
  • 20:36 dancy@deploy1003: jforrester, dancy: Continuing with deployment
  • 20:35 dancy@deploy1003: jforrester, dancy: Backport for wikisource-ocr-langs: Lower timeout, cache absence, set lockTSE+busyValue (T442064) (1.47.0-wmf.24) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 20:34 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1013.eqiad.wmnet
  • 20:34 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1013.eqiad.wmnet
  • 20:34 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1012.eqiad.wmnet
  • 20:34 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1012.eqiad.wmnet
  • 20:33 dancy@deploy1003: Started scap sync-world: Backport for wikisource-ocr-langs: Lower timeout, cache absence, set lockTSE+busyValue (T442064) (1.47.0-wmf.24)
  • 20:30 thcipriani@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: Deployments locked for the week (duration: 156m 22s)
  • 20:30 dancy@deploy1003: Forcefully removing global lock: Deployments locked for the week
  • 20:29 dancy@deploy1003: Forcefully removing global lock: Deployments locked for the week
  • 20:28 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1012.eqiad.wmnet
  • 20:17 rzl@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for phab2003.codfw.wmnet,phab1005.eqiad.wmnet
  • 20:17 rzl@cumin1004: START - Cookbook sre.hosts.remove-downtime for phab2003.codfw.wmnet,phab1005.eqiad.wmnet
  • 20:10 thcipriani: phab restart
  • 20:10 rzl@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on phab2003.codfw.wmnet,phab1005.eqiad.wmnet with reason: deploying
  • 19:58 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1012.eqiad.wmnet
  • 19:57 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1011.eqiad.wmnet
  • 19:57 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1011.eqiad.wmnet
  • 19:51 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1011.eqiad.wmnet
  • 19:39 brett@cumin2003: END (PASS) - Cookbook sre.cdn.roll-upgrade-ats (exit_code=0) Rolling upgrade of ATS on P{cp5017.eqsin.wmnet} and A:cp - 9.2.16 upgrade (T441945)
  • 19:39 brett@cumin2003: END (PASS) - Cookbook sre.cdn.roll-upgrade-ats (exit_code=0) Rolling upgrade of ATS on P{cp5032.eqsin.wmnet} and A:cp - 9.2.16 upgrade (T441945)
  • 19:33 brett@cumin2003: START - Cookbook sre.cdn.roll-upgrade-ats Rolling upgrade of ATS on P{cp5032.eqsin.wmnet} and A:cp - 9.2.16 upgrade (T441945)
  • 19:33 brett@cumin2003: START - Cookbook sre.cdn.roll-upgrade-ats Rolling upgrade of ATS on P{cp5017.eqsin.wmnet} and A:cp - 9.2.16 upgrade (T441945)
  • 19:27 rzl@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 19:26 rzl@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 19:26 rzl@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 19:26 rzl@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 19:21 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1011.eqiad.wmnet
  • 19:21 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1010.eqiad.wmnet
  • 19:21 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1010.eqiad.wmnet
  • 19:15 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1010.eqiad.wmnet
  • 19:01 rzl@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 18:59 rzl@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 18:55 rzl@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 18:53 rzl@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 18:45 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1010.eqiad.wmnet
  • 18:45 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1009.eqiad.wmnet
  • 18:45 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1009.eqiad.wmnet
  • 18:39 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1009.eqiad.wmnet
  • 20:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1009.eqiad.wmnet
  • 20:08 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1008.eqiad.wmnet
  • 20:08 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1008.eqiad.wmnet
  • 20:01 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1008.eqiad.wmnet
  • 20:01 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1008.eqiad.wmnet
  • 20:01 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1007.eqiad.wmnet
  • 20:00 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1007.eqiad.wmnet
  • 19:54 mutante: codesearch zombie hound procs piling up. codesearch1001 ran out of PIDs.
  • 19:54 thcipriani@deploy1003: Locking from deployment [ALL REPOSITORIES]: Deployments locked for the week
  • 19:53 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1007.eqiad.wmnet
  • 19:53 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1007.eqiad.wmnet
  • 19:53 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1006.eqiad.wmnet
  • 19:53 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1006.eqiad.wmnet
  • 19:50 ladsgroup@deploy1003: Finished scap sync-world: Backport for Upgrading composer/ca-bundle (1.5.14 => 1.6.0) (1.47.0-wmf.24) (duration: 43m 35s)
  • 19:46 mutante: codesearch1001 - hard rebooting at ganeti level
  • 19:45 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp5017.*
  • 19:42 ladsgroup@deploy1003: ladsgroup: Continuing with deployment
  • 19:41 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1006.eqiad.wmnet
  • 19:41 ladsgroup@deploy1003: ladsgroup: Backport for Upgrading composer/ca-bundle (1.5.14 => 1.6.0) (1.47.0-wmf.24) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 19:41 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 19:41 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 19:40 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 19:40 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 19:29 dzahn@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on codesearch2001.codfw.wmnet,codesearch1001.eqiad.wmnet with reason: unknown
  • 19:12 brett@cumin2003: END (PASS) - Cookbook sre.cdn.roll-upgrade-ats (exit_code=0) Rolling upgrade of ATS on A:cp-text_magru and not P{cp7001.magru.wmnet} and A:cp - 9.2.16 upgrade (T441945)
  • 19:11 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1006.eqiad.wmnet
  • 19:11 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1005.eqiad.wmnet
  • 19:11 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1005.eqiad.wmnet
  • 19:09 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-test-client1002.eqiad.wmnet
  • 19:06 ladsgroup@deploy1003: Started scap sync-world: Backport for Upgrading composer/ca-bundle (1.5.14 => 1.6.0) (1.47.0-wmf.24)
  • 19:04 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host an-test-client1002.eqiad.wmnet
  • 19:00 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1005.eqiad.wmnet
  • 18:55 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1005.eqiad.wmnet
  • 18:55 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1004.eqiad.wmnet
  • 18:55 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1004.eqiad.wmnet
  • 18:53 eevans@deploy1003: helmfile [codfw] DONE helmfile.d/services/linked-artifacts: apply
  • 18:53 eevans@deploy1003: helmfile [codfw] START helmfile.d/services/linked-artifacts: apply
  • 18:53 eevans@deploy1003: helmfile [eqiad] DONE helmfile.d/services/linked-artifacts: apply
  • 18:52 eevans@deploy1003: helmfile [eqiad] START helmfile.d/services/linked-artifacts: apply
  • 18:49 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1004.eqiad.wmnet
  • 18:46 brett@cumin2003: END (PASS) - Cookbook sre.cdn.roll-upgrade-ats (exit_code=0) Rolling upgrade of ATS on A:cp-upload_magru and not P{cp7016.magru.wmnet} and A:cp - 9.2.16 upgrade (T441945)
  • 18:45 claime: homer "cr*eqiad" commit 'failed wikikube-worker1152'
  • 18:43 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply
  • 18:43 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply
  • 18:36 fabfur@cumin1004: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on A:cp-text_eqiad
  • 18:36 fabfur@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp1114.eqiad.wmnet
  • 18:36 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-launcher1003.eqiad.wmnet
  • 18:29 cgoubert@dns2004: END - running authdns-update
  • 18:29 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host an-launcher1003.eqiad.wmnet
  • 18:26 cgoubert@dns2004: START - running authdns-update
  • 18:20 sukhe@cumin1004: END (PASS) - Cookbook sre.dns.roll-reboot (exit_code=0) rolling reboot on A:eqiad and (A:dnsbox)
  • 18:20 sukhe@cumin1004: cookbooks.sre.dns.roll-reboot finished rebooting dns1006.wikimedia.org
  • 18:19 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1004.eqiad.wmnet
  • 18:19 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1003.eqiad.wmnet
  • 18:19 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1003.eqiad.wmnet
  • 18:17 brett@cumin2003: START - Cookbook sre.cdn.roll-upgrade-ats Rolling upgrade of ATS on A:cp-upload_magru and not P{cp7016.magru.wmnet} and A:cp - 9.2.16 upgrade (T441945)
  • 18:17 brett@cumin2003: START - Cookbook sre.cdn.roll-upgrade-ats Rolling upgrade of ATS on A:cp-text_magru and not P{cp7001.magru.wmnet} and A:cp - 9.2.16 upgrade (T441945)
  • 18:12 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1003.eqiad.wmnet
  • 18:12 sukhe@cumin1004: END (PASS) - Cookbook sre.dns.roll-reboot (exit_code=0) rolling reboot on A:codfw and (A:dnsbox)
  • 18:12 sukhe@cumin1004: cookbooks.sre.dns.roll-reboot finished rebooting dns2006.wikimedia.org
  • 18:12 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1003.eqiad.wmnet
  • 18:12 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1002.eqiad.wmnet
  • 18:12 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1002.eqiad.wmnet
  • 18:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1009.eqiad.wmnet
  • 18:08 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1008.eqiad.wmnet
  • 18:08 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1008.eqiad.wmnet
  • 18:06 sukhe@cumin1004: cookbooks.sre.dns.roll-reboot begin reboot of dns1006.wikimedia.org
  • 18:05 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1002.eqiad.wmnet
  • 18:01 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1008.eqiad.wmnet
  • 18:01 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1008.eqiad.wmnet
  • 18:01 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1007.eqiad.wmnet
  • 18:00 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1007.eqiad.wmnet
  • 17:58 sukhe@cumin1004: cookbooks.sre.dns.roll-reboot begin reboot of dns2006.wikimedia.org
  • 17:57 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2091.codfw.wmnet
  • 17:57 fabfur@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp1112.eqiad.wmnet
  • 17:56 mvernon@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1091.eqiad.wmnet
  • 17:54 mutante: codesearch zombie hound procs piling up. codesearch1001 ran out of PIDs.
  • 17:54 thcipriani@deploy1003: Locking from deployment [ALL REPOSITORIES]: Deployments locked for the week
  • 17:53 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1007.eqiad.wmnet
  • 17:53 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1007.eqiad.wmnet
  • 17:53 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1006.eqiad.wmnet
  • 17:53 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1006.eqiad.wmnet
  • 17:51 sukhe@cumin1004: cookbooks.sre.dns.roll-reboot finished rebooting dns1005.wikimedia.org
  • 17:50 mvernon@cumin2003: START - Cookbook sre.hosts.reboot-single for host ms-be2091.codfw.wmnet
  • 17:50 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2090.codfw.wmnet
  • 17:50 ladsgroup@deploy1003: Finished scap sync-world: Backport for Upgrading composer/ca-bundle (1.5.14 => 1.6.0) (1.47.0-wmf.24) (duration: 43m 35s)
  • 17:49 mvernon@cumin1004: START - Cookbook sre.hosts.reboot-single for host ms-be1091.eqiad.wmnet
  • 17:48 mvernon@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1090.eqiad.wmnet
  • 17:46 mutante: codesearch1001 - hard rebooting at ganeti level
  • 17:45 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp5017.*
  • 17:44 mvernon@cumin2003: START - Cookbook sre.hosts.reboot-single for host ms-be2090.codfw.wmnet
  • 17:43 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2089.codfw.wmnet
  • 17:43 sukhe@cumin1004: cookbooks.sre.dns.roll-reboot finished rebooting dns2005.wikimedia.org
  • 17:42 sukhe@cumin1004: END (PASS) - Cookbook sre.dns.roll-reboot (exit_code=0) rolling reboot on A:esams and (A:dnsbox)
  • 17:42 sukhe@cumin1004: cookbooks.sre.dns.roll-reboot finished rebooting dns3004.wikimedia.org
  • 17:42 mvernon@cumin1004: START - Cookbook sre.hosts.reboot-single for host ms-be1090.eqiad.wmnet
  • 17:42 mvernon@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1089.eqiad.wmnet
  • 17:40 kamila@deploy1003: Locking from deployment [ALL REPOSITORIES]: Deployments locked for the week
  • 17:38 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/lac-lambdas: apply
  • 17:38 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/lac-lambdas: apply
  • 17:38 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/lac-lambdas: apply
  • 17:38 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/lac-lambdas: apply
  • 17:37 mvernon@cumin2003: START - Cookbook sre.hosts.reboot-single for host ms-be2089.codfw.wmnet
  • 17:37 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2088.codfw.wmnet
  • 17:36 kamila@deploy1003: Finished scap build-images: T441933 (duration: 11m 28s)
  • 17:35 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-test-ui1001.eqiad.wmnet
  • 17:35 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/lac-lambdas: apply
  • 17:35 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/lac-lambdas: apply
  • 17:35 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1002.eqiad.wmnet
  • 17:35 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1001.eqiad.wmnet
  • 17:35 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1001.eqiad.wmnet
  • 17:35 mvernon@cumin1004: START - Cookbook sre.hosts.reboot-single for host ms-be1089.eqiad.wmnet
  • 17:34 mvernon@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1088.eqiad.wmnet
  • 17:33 sukhe@cumin1004: cookbooks.sre.dns.roll-reboot begin reboot of dns3004.wikimedia.org
  • 17:32 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host an-test-ui1001.eqiad.wmnet
  • 17:31 sukhe@cumin1004: cookbooks.sre.dns.roll-reboot begin reboot of dns1005.wikimedia.org
  • 17:30 mvernon@cumin2003: START - Cookbook sre.hosts.reboot-single for host ms-be2088.codfw.wmnet
  • 17:30 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2087.codfw.wmnet
  • 17:29 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-tool1008.eqiad.wmnet
  • 17:29 sukhe@cumin1004: cookbooks.sre.dns.roll-reboot begin reboot of dns2005.wikimedia.org
  • 17:28 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1001.eqiad.wmnet
  • 17:27 mvernon@cumin1004: START - Cookbook sre.hosts.reboot-single for host ms-be1088.eqiad.wmnet
  • 17:27 mvernon@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1087.eqiad.wmnet
  • 17:25 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host an-tool1008.eqiad.wmnet
  • 17:24 cmooney@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device cloudsw3-d5-eqiad
  • 17:24 kamila@deploy1003: Started scap build-images: T441933
  • 17:24 cmooney@cumin1004: START - Cookbook sre.network.tls for network device cloudsw3-d5-eqiad
  • 17:24 dancy@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: Deployments locked for the week (duration: 1211m 57s)
  • 17:24 kamila@deploy1003: Forcefully removing global lock: rebuild images, no deploy
  • 17:23 mvernon@cumin2003: START - Cookbook sre.hosts.reboot-single for host ms-be2087.codfw.wmnet
  • 17:23 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2086.codfw.wmnet
  • 17:18 mvernon@cumin1004: START - Cookbook sre.hosts.reboot-single for host ms-be1087.eqiad.wmnet
  • 17:18 mvernon@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1086.eqiad.wmnet
  • 17:18 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host stat1010.eqiad.wmnet
  • 17:18 sukhe@cumin1004: cookbooks.sre.dns.roll-reboot finished rebooting dns3003.wikimedia.org
  • 17:17 mvernon@cumin2003: START - Cookbook sre.hosts.reboot-single for host ms-be2086.codfw.wmnet
  • 17:17 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2085.codfw.wmnet
  • 17:17 fabfur@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp1110.eqiad.wmnet
  • 17:17 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host stat1009.eqiad.wmnet
  • 17:16 sukhe@cumin1004: cookbooks.sre.dns.roll-reboot finished rebooting dns1004.wikimedia.org
  • 17:16 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host stat1011.eqiad.wmnet
  • 17:16 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host stat1008.eqiad.wmnet
  • 17:14 sukhe@cumin1004: cookbooks.sre.dns.roll-reboot finished rebooting dns2004.wikimedia.org
  • 17:12 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/turnilo: apply
  • 17:12 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/turnilo: apply
  • 17:11 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/turnilo: apply
  • 17:11 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/turnilo: apply
  • 17:11 mvernon@cumin2003: START - Cookbook sre.hosts.reboot-single for host ms-be2085.codfw.wmnet
  • 17:11 mvernon@cumin1004: START - Cookbook sre.hosts.reboot-single for host ms-be1086.eqiad.wmnet
  • 17:11 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2084.codfw.wmnet
  • 17:11 mvernon@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1085.eqiad.wmnet
  • 17:08 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host stat1011.eqiad.wmnet
  • 17:08 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host stat1010.eqiad.wmnet
  • 17:07 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host stat1009.eqiad.wmnet
  • 17:07 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host stat1008.eqiad.wmnet
  • 17:04 mvernon@cumin2003: START - Cookbook sre.hosts.reboot-single for host ms-be2084.codfw.wmnet
  • 17:04 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2083.codfw.wmnet
  • 17:04 mvernon@cumin1004: START - Cookbook sre.hosts.reboot-single for host ms-be1085.eqiad.wmnet
  • 17:04 mvernon@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1084.eqiad.wmnet
  • 17:00 sukhe@cumin1004: cookbooks.sre.dns.roll-reboot begin reboot of dns3003.wikimedia.org
  • 17:00 sukhe@cumin1004: START - Cookbook sre.dns.roll-reboot rolling reboot on A:esams and (A:dnsbox)
  • 17:00 sukhe@cumin1004: cookbooks.sre.dns.roll-reboot begin reboot of dns2004.wikimedia.org
  • 17:00 sukhe@cumin1004: START - Cookbook sre.dns.roll-reboot rolling reboot on A:codfw and (A:dnsbox)
  • 17:00 sukhe@cumin1004: cookbooks.sre.dns.roll-reboot begin reboot of dns1004.wikimedia.org
  • 17:00 sukhe@cumin1004: START - Cookbook sre.dns.roll-reboot rolling reboot on A:eqiad and (A:dnsbox)
  • 16:58 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1001.eqiad.wmnet
  • 16:58 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs-test1001.eqiad.wmnet
  • 16:58 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs-test1001.eqiad.wmnet
  • 16:57 mvernon@cumin2003: START - Cookbook sre.hosts.reboot-single for host ms-be2083.codfw.wmnet
  • 16:57 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2082.codfw.wmnet
  • 16:57 mvernon@cumin1004: START - Cookbook sre.hosts.reboot-single for host ms-be1084.eqiad.wmnet
  • 16:57 mvernon@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1083.eqiad.wmnet
  • 16:51 papaul: ongoing maintenance on mr1-eqsin
  • 16:50 pt1979@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 10 hosts with reason: Junos upgrade
  • 16:50 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs-test1001.eqiad.wmnet
  • 16:50 mvernon@cumin2003: START - Cookbook sre.hosts.reboot-single for host ms-be2082.codfw.wmnet
  • 16:50 mvernon@cumin1004: START - Cookbook sre.hosts.reboot-single for host ms-be1083.eqiad.wmnet
  • 16:50 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2081.codfw.wmnet
  • 16:50 mvernon@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1082.eqiad.wmnet
  • 16:49 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs-test1001.eqiad.wmnet
  • 16:49 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1003.eqiad.wmnet
  • 16:49 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1003.eqiad.wmnet
  • 16:48 pt1979@cumin2003: DONE (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on 11 hosts with reason: Junos upgrade
  • 16:45 pt1979@cumin2003: DONE (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on 11 hosts with reason: Junos upgrade
  • 16:44 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1003.eqiad.wmnet
  • 16:43 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1003.eqiad.wmnet
  • 16:43 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1002.eqiad.wmnet
  • 16:43 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1002.eqiad.wmnet
  • 16:43 mvernon@cumin2003: START - Cookbook sre.hosts.reboot-single for host ms-be2081.codfw.wmnet
  • 16:43 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2080.codfw.wmnet
  • 16:42 mvernon@cumin1004: START - Cookbook sre.hosts.reboot-single for host ms-be1082.eqiad.wmnet
  • 16:42 mvernon@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1081.eqiad.wmnet
  • 16:38 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1002.eqiad.wmnet
  • 16:37 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1002.eqiad.wmnet
  • 16:37 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1001.eqiad.wmnet
  • 16:37 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1001.eqiad.wmnet
  • 16:35 mvernon@cumin2003: START - Cookbook sre.hosts.reboot-single for host ms-be2080.codfw.wmnet
  • 16:35 bking@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply
  • 16:35 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2079.codfw.wmnet
  • 16:35 bking@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply
  • 16:35 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply
  • 16:35 fabfur@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp1108.eqiad.wmnet
  • 16:35 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply
  • 16:35 mvernon@cumin1004: START - Cookbook sre.hosts.reboot-single for host ms-be1081.eqiad.wmnet
  • 16:35 bking@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-ttmserver: apply
  • 16:35 mvernon@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1080.eqiad.wmnet
  • 16:34 bking@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-ttmserver: apply
  • 16:34 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-ttmserver: apply
  • 16:34 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-ttmserver: apply
  • 16:34 bking@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-toolhub-test: apply
  • 16:34 bking@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-toolhub-test: apply
  • 16:34 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-toolhub-test: apply
  • 16:34 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-toolhub-test: apply
  • 16:34 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1001.eqiad.wmnet
  • 16:33 bking@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-toolhub: apply
  • 16:33 bking@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-toolhub: apply
  • 16:33 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-toolhub: apply
  • 16:33 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-toolhub: apply
  • 16:33 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1001.eqiad.wmnet
  • 16:33 bking@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-test: apply
  • 16:33 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:dse-k8s-worker-eqiad
  • 16:33 bking@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-test: apply
  • 16:33 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-test: apply
  • 16:33 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-test: apply
  • 16:32 bking@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply
  • 16:32 bking@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply
  • 16:32 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply
  • 16:32 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply
  • 16:32 bking@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-ipoid-test: apply
  • 16:32 bking@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-ipoid-test: apply
  • 16:32 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbprov2007.codfw.wmnet with OS trixie
  • 16:31 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-ipoid-test: apply
  • 16:31 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-ipoid-test: apply
  • 16:27 mvernon@cumin1004: START - Cookbook sre.hosts.reboot-single for host ms-be1080.eqiad.wmnet
  • 16:27 mvernon@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1079.eqiad.wmnet
  • 16:26 mvernon@cumin2003: START - Cookbook sre.hosts.reboot-single for host ms-be2079.codfw.wmnet
  • 16:26 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2078.codfw.wmnet
  • 16:26 cmooney@dns3003: END - running authdns-update
  • 16:25 cmooney@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device cloudsw3-c8-eqiad
  • 16:24 cmooney@cumin1004: START - Cookbook sre.network.tls for network device cloudsw3-c8-eqiad
  • 16:23 cmooney@dns3003: START - running authdns-update
  • 16:21 bking@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-ipoid: apply
  • 16:20 bking@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-ipoid: apply
  • 16:20 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-ipoid: apply
  • 16:20 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-ipoid: apply
  • 16:19 bking@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-apifeatureusage-test: apply
  • 16:19 mvernon@cumin1004: START - Cookbook sre.hosts.reboot-single for host ms-be1079.eqiad.wmnet
  • 16:19 bking@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-apifeatureusage-test: apply
  • 16:19 mvernon@cumin2003: START - Cookbook sre.hosts.reboot-single for host ms-be2078.codfw.wmnet
  • 16:19 mvernon@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1078.eqiad.wmnet
  • 16:19 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2077.codfw.wmnet
  • 16:19 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-apifeatureusage-test: apply
  • 16:19 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-apifeatureusage-test: apply
  • 16:18 bking@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-apifeatureusage: apply
  • 16:18 bking@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-apifeatureusage: apply
  • 16:17 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbprov2007.codfw.wmnet with reason: host reimage
  • 16:16 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-apifeatureusage: apply
  • 16:16 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-apifeatureusage: apply
  • 16:14 marostegui@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dbprov2007.codfw.wmnet with reason: host reimage
  • 16:12 mvernon@cumin2003: START - Cookbook sre.hosts.reboot-single for host ms-be2077.codfw.wmnet
  • 16:12 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2076.codfw.wmnet
  • 16:12 mvernon@cumin1004: START - Cookbook sre.hosts.reboot-single for host ms-be1078.eqiad.wmnet
  • 16:12 mvernon@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1077.eqiad.wmnet
  • 16:10 slyngshede@cumin1004: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on A:cp-upload_eqsin and A:cp
  • 16:10 slyngshede@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp5032.eqsin.wmnet
  • 16:06 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 16:06 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: remove entries for last eqord link - cmooney@cumin1004"
  • 16:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2227 (T441633)', diff saved to https://phabricator.wikimedia.org/P96980 and previous config saved to /var/cache/conftool/dbconfig/20261022-140508-ladsgroup.json
  • 16:04 mvernon@cumin2003: START - Cookbook sre.hosts.reboot-single for host ms-be2076.codfw.wmnet
  • 16:04 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2075.codfw.wmnet
  • 16:04 mvernon@cumin1004: START - Cookbook sre.hosts.reboot-single for host ms-be1077.eqiad.wmnet
  • 16:04 mvernon@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1076.eqiad.wmnet
  • 16:03 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: remove entries for last eqord link - cmooney@cumin1004"
  • 16:02 slyngshede@cumin1004: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P{cp5024*} and A:cp
  • 16:02 slyngshede@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp5024.eqsin.wmnet
  • 16:00 marostegui@cumin1004: START - Cookbook sre.hosts.reimage for host dbprov2007.codfw.wmnet with OS trixie
  • 15:58 cmooney@cumin1004: START - Cookbook sre.dns.netbox
  • 15:57 mvernon@cumin2003: START - Cookbook sre.hosts.reboot-single for host ms-be2075.codfw.wmnet
  • 15:57 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2074.codfw.wmnet
  • 15:56 fabfur@cumin1004: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on A:cp-upload_eqiad
  • 15:56 fabfur@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp1115.eqiad.wmnet
  • 15:55 mvernon@cumin1004: START - Cookbook sre.hosts.reboot-single for host ms-be1076.eqiad.wmnet
  • 15:55 mvernon@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1075.eqiad.wmnet
  • 15:55 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2227', diff saved to https://phabricator.wikimedia.org/P96979 and previous config saved to /var/cache/conftool/dbconfig/20261022-135506-ladsgroup.json
  • 15:53 fabfur@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp1106.eqiad.wmnet
  • 15:52 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.sync-bgp-netbox (exit_code=0)
  • 15:51 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.sync-bgp-netbox (exit_code=0)
  • 15:51 ayounsi@cumin1004: START - Cookbook sre.network.sync-bgp-netbox
  • 15:50 slyngshede@cumin1004: START - Cookbook sre.cdn.roll-reboot rolling reboot on P{cp5024*} and A:cp
  • 15:50 mvernon@cumin2003: START - Cookbook sre.hosts.reboot-single for host ms-be2074.codfw.wmnet
  • 15:50 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2073.codfw.wmnet
  • 15:48 marostegui@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host dbprov2007.codfw.wmnet with OS trixie
  • 15:47 mvernon@cumin1004: START - Cookbook sre.hosts.reboot-single for host ms-be1075.eqiad.wmnet
  • 15:47 mvernon@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1074.eqiad.wmnet
  • 15:47 slyngshede@cumin1004: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P{cp5022*} OR P{cp5023*} and A:cp
  • 15:47 slyngshede@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp5023.eqsin.wmnet
  • 15:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2227', diff saved to https://phabricator.wikimedia.org/P96978 and previous config saved to /var/cache/conftool/dbconfig/20261022-134504-ladsgroup.json
  • 15:42 mvernon@cumin2003: START - Cookbook sre.hosts.reboot-single for host ms-be2073.codfw.wmnet
  • 15:42 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2072.codfw.wmnet
  • 15:37 mvernon@cumin1004: START - Cookbook sre.hosts.reboot-single for host ms-be1074.eqiad.wmnet
  • 15:37 mvernon@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1073.eqiad.wmnet
  • 15:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2227 (T441633)', diff saved to https://phabricator.wikimedia.org/P96977 and previous config saved to /var/cache/conftool/dbconfig/20261022-133502-ladsgroup.json
  • 15:34 mvernon@cumin2003: START - Cookbook sre.hosts.reboot-single for host ms-be2072.codfw.wmnet
  • 15:34 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2071.codfw.wmnet
  • 15:33 marostegui@cumin1004: START - Cookbook sre.hosts.reimage for host dbprov2007.codfw.wmnet with OS trixie
  • 15:29 mvernon@cumin1004: START - Cookbook sre.hosts.reboot-single for host ms-be1073.eqiad.wmnet
  • 15:29 mvernon@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1072.eqiad.wmnet
  • 15:28 slyngshede@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp5031.eqsin.wmnet
  • 15:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2227 (T441633)', diff saved to https://phabricator.wikimedia.org/P96976 and previous config saved to /var/cache/conftool/dbconfig/20261022-132739-ladsgroup.json
  • 15:27 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2227.codfw.wmnet with reason: Maintenance
  • 15:27 mvernon@cumin2003: START - Cookbook sre.hosts.reboot-single for host ms-be2071.codfw.wmnet
  • 15:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2209 (T441633)', diff saved to https://phabricator.wikimedia.org/P96975 and previous config saved to /var/cache/conftool/dbconfig/20261022-132718-ladsgroup.json
  • 15:27 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2070.codfw.wmnet
  • 15:21 mvernon@cumin1004: START - Cookbook sre.hosts.reboot-single for host ms-be1072.eqiad.wmnet
  • 15:21 mvernon@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1071.eqiad.wmnet
  • 15:19 mvernon@cumin2003: START - Cookbook sre.hosts.reboot-single for host ms-be2070.codfw.wmnet
  • 15:19 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2069.codfw.wmnet
  • 15:17 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2209', diff saved to https://phabricator.wikimedia.org/P96974 and previous config saved to /var/cache/conftool/dbconfig/20261022-131716-ladsgroup.json
  • 15:16 fabfur@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp1113.eqiad.wmnet
  • 15:13 mvernon@cumin1004: START - Cookbook sre.hosts.reboot-single for host ms-be1071.eqiad.wmnet
  • 15:13 mvernon@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1070.eqiad.wmnet
  • 15:13 fabfur@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp1104.eqiad.wmnet
  • 15:11 mvernon@cumin2003: START - Cookbook sre.hosts.reboot-single for host ms-be2069.codfw.wmnet
  • 15:11 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2068.codfw.wmnet
  • 15:07 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2209', diff saved to https://phabricator.wikimedia.org/P96973 and previous config saved to /var/cache/conftool/dbconfig/20261022-130714-ladsgroup.json
  • 15:05 mvernon@cumin1004: START - Cookbook sre.hosts.reboot-single for host ms-be1070.eqiad.wmnet
  • 15:05 mvernon@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1069.eqiad.wmnet
  • 15:05 slyngshede@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp5022.eqsin.wmnet
  • 15:03 mvernon@cumin2003: START - Cookbook sre.hosts.reboot-single for host ms-be2068.codfw.wmnet
  • 15:03 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2067.codfw.wmnet
  • 15:02 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbprov2007.codfw.wmnet with OS bookworm
  • 14:57 mvernon@cumin1004: START - Cookbook sre.hosts.reboot-single for host ms-be1069.eqiad.wmnet
  • 14:57 mvernon@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1068.eqiad.wmnet
  • 14:57 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2209 (T441633)', diff saved to https://phabricator.wikimedia.org/P96972 and previous config saved to /var/cache/conftool/dbconfig/20261022-125713-ladsgroup.json
  • 14:56 mvernon@cumin2003: START - Cookbook sre.hosts.reboot-single for host ms-be2067.codfw.wmnet
  • 14:56 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2066.codfw.wmnet
  • 14:53 slyngshede@cumin1004: START - Cookbook sre.cdn.roll-reboot rolling reboot on P{cp5022*} OR P{cp5023*} and A:cp
  • 14:50 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2209 (T441633)', diff saved to https://phabricator.wikimedia.org/P96971 and previous config saved to /var/cache/conftool/dbconfig/20261022-125005-ladsgroup.json
  • 14:50 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2209.codfw.wmnet with reason: Maintenance
  • 14:49 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2194 (T441633)', diff saved to https://phabricator.wikimedia.org/P96970 and previous config saved to /var/cache/conftool/dbconfig/20261022-124943-ladsgroup.json
  • 14:48 mvernon@cumin2003: START - Cookbook sre.hosts.reboot-single for host ms-be2066.codfw.wmnet
  • 14:48 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2065.codfw.wmnet
  • 14:48 slyngshede@cumin1004: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P{cp5018*} OR P{cp5019*} OR P{cp5020*} OR P{cp5021*} and A:cp
  • 14:48 slyngshede@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp5021.eqsin.wmnet
  • 14:47 mvernon@cumin1004: START - Cookbook sre.hosts.reboot-single for host ms-be1068.eqiad.wmnet
  • 14:47 mvernon@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1067.eqiad.wmnet
  • 14:46 slyngshede@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp5030.eqsin.wmnet
  • 14:44 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbprov2007.codfw.wmnet with reason: host reimage
  • 14:41 marostegui@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dbprov2007.codfw.wmnet with reason: host reimage
  • 14:40 mvernon@cumin2003: START - Cookbook sre.hosts.reboot-single for host ms-be2065.codfw.wmnet
  • 14:40 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2064.codfw.wmnet
  • 14:40 mvernon@cumin1004: START - Cookbook sre.hosts.reboot-single for host ms-be1067.eqiad.wmnet
  • 14:40 mvernon@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1066.eqiad.wmnet
  • 14:39 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2194', diff saved to https://phabricator.wikimedia.org/P96968 and previous config saved to /var/cache/conftool/dbconfig/20261022-123941-ladsgroup.json
  • 14:36 fabfur@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp1111.eqiad.wmnet
  • 14:33 fabfur@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp1103.eqiad.wmnet
  • 14:32 mvernon@cumin1004: START - Cookbook sre.hosts.reboot-single for host ms-be1066.eqiad.wmnet
  • 14:31 mvernon@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1065.eqiad.wmnet
  • 14:30 mvernon@cumin2003: START - Cookbook sre.hosts.reboot-single for host ms-be2064.codfw.wmnet
  • 14:30 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2063.codfw.wmnet
  • 14:29 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2194', diff saved to https://phabricator.wikimedia.org/P96967 and previous config saved to /var/cache/conftool/dbconfig/20261022-122939-ladsgroup.json
  • 14:24 mvernon@cumin1004: START - Cookbook sre.hosts.reboot-single for host ms-be1065.eqiad.wmnet
  • 14:23 mvernon@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1064.eqiad.wmnet
  • 14:23 marostegui@cumin1004: START - Cookbook sre.hosts.reimage for host dbprov2007.codfw.wmnet with OS bookworm
  • 14:23 mvernon@cumin2003: START - Cookbook sre.hosts.reboot-single for host ms-be2063.codfw.wmnet
  • 14:23 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2062.codfw.wmnet
  • 14:19 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2194 (T441633)', diff saved to https://phabricator.wikimedia.org/P96966 and previous config saved to /var/cache/conftool/dbconfig/20261022-121938-ladsgroup.json
  • 14:16 marostegui@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host dbprov2007.codfw.wmnet with OS trixie
  • 14:16 mvernon@cumin2003: START - Cookbook sre.hosts.reboot-single for host ms-be2062.codfw.wmnet
  • 14:15 mvernon@cumin1004: START - Cookbook sre.hosts.reboot-single for host ms-be1064.eqiad.wmnet
  • 14:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2194 (T441633)', diff saved to https://phabricator.wikimedia.org/P96964 and previous config saved to /var/cache/conftool/dbconfig/20261022-121110-ladsgroup.json
  • 14:11 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2194.codfw.wmnet with reason: Maintenance
  • 14:10 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2190 (T441633)', diff saved to https://phabricator.wikimedia.org/P96963 and previous config saved to /var/cache/conftool/dbconfig/20261022-121048-ladsgroup.json
  • 14:06 slyngshede@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp5020.eqsin.wmnet
  • 14:04 slyngshede@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp5029.eqsin.wmnet
  • 14:00 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2190', diff saved to https://phabricator.wikimedia.org/P96962 and previous config saved to /var/cache/conftool/dbconfig/20261022-120046-ladsgroup.json
  • 14:00 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
  • 14:00 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
  • 13:56 fabfur@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp1109.eqiad.wmnet
  • 13:54 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:dse-k8s-worker-codfw
  • 13:54 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2011.codfw.wmnet
  • 13:54 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2011.codfw.wmnet
  • 13:54 fabfur@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp1102.eqiad.wmnet
  • 13:50 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2190', diff saved to https://phabricator.wikimedia.org/P96960 and previous config saved to /var/cache/conftool/dbconfig/20261022-115044-ladsgroup.json
  • 13:48 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2011.codfw.wmnet
  • 13:48 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2011.codfw.wmnet
  • 13:47 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2010.codfw.wmnet
  • 13:47 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2010.codfw.wmnet
  • 13:42 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2010.codfw.wmnet
  • 13:41 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2010.codfw.wmnet
  • 13:41 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2009.codfw.wmnet
  • 13:41 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2009.codfw.wmnet
  • 13:40 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2190 (T441633)', diff saved to https://phabricator.wikimedia.org/P96959 and previous config saved to /var/cache/conftool/dbconfig/20261022-114042-ladsgroup.json
  • 13:35 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2009.codfw.wmnet
  • 13:35 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2009.codfw.wmnet
  • 13:35 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2008.codfw.wmnet
  • 13:35 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2008.codfw.wmnet
  • 13:32 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2190 (T441633)', diff saved to https://phabricator.wikimedia.org/P96958 and previous config saved to /var/cache/conftool/dbconfig/20261022-113217-ladsgroup.json
  • 13:32 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2190.codfw.wmnet with reason: Maintenance
  • 13:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2177 (T441633)', diff saved to https://phabricator.wikimedia.org/P96957 and previous config saved to /var/cache/conftool/dbconfig/20261022-113155-ladsgroup.json
  • 13:29 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2008.codfw.wmnet
  • 13:29 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2008.codfw.wmnet
  • 13:28 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2007.codfw.wmnet
  • 13:28 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2007.codfw.wmnet
  • 13:24 slyngshede@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp5019.eqsin.wmnet
  • 13:23 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2007.codfw.wmnet
  • 13:23 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2007.codfw.wmnet
  • 13:22 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2006.codfw.wmnet
  • 13:22 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2006.codfw.wmnet
  • 13:22 slyngshede@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp5028.eqsin.wmnet
  • 13:22 marostegui@cumin1004: START - Cookbook sre.hosts.reimage for host dbprov2007.codfw.wmnet with OS trixie
  • 13:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2177', diff saved to https://phabricator.wikimedia.org/P96956 and previous config saved to /var/cache/conftool/dbconfig/20261022-112153-ladsgroup.json
  • 13:21 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-test-coord1002.eqiad.wmnet
  • 13:18 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2006.codfw.wmnet
  • 13:18 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2006.codfw.wmnet
  • 13:18 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2005.codfw.wmnet
  • 13:18 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2005.codfw.wmnet
  • 13:17 dkertesz@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-text_esams and A:cp - upgrade haproxy to 3.2.26-1 (T441435)
  • 13:16 fabfur@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp1107.eqiad.wmnet
  • 13:14 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host an-test-coord1002.eqiad.wmnet
  • 13:13 fabfur@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp1101.eqiad.wmnet
  • 13:13 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-test-master1003.eqiad.wmnet
  • 13:12 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2005.codfw.wmnet
  • 13:12 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2005.codfw.wmnet
  • 13:12 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2004.codfw.wmnet
  • 13:12 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2004.codfw.wmnet
  • 13:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2177', diff saved to https://phabricator.wikimedia.org/P96955 and previous config saved to /var/cache/conftool/dbconfig/20261022-111151-ladsgroup.json
  • 13:06 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2004.codfw.wmnet
  • 13:06 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host an-test-master1003.eqiad.wmnet
  • 13:01 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2177 (T441633)', diff saved to https://phabricator.wikimedia.org/P96954 and previous config saved to /var/cache/conftool/dbconfig/20261022-110149-ladsgroup.json
  • 13:01 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2004.codfw.wmnet
  • 13:01 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2003.codfw.wmnet
  • 13:01 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2003.codfw.wmnet
  • 12:59 btullis@dns1004: END - running authdns-update
  • 12:57 btullis@dns1004: START - running authdns-update
  • 12:56 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2003.codfw.wmnet
  • 12:52 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2177 (T441633)', diff saved to https://phabricator.wikimedia.org/P96952 and previous config saved to /var/cache/conftool/dbconfig/20261022-105226-ladsgroup.json
  • 12:52 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2177.codfw.wmnet with reason: Maintenance
  • 12:52 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2156 (T441633)', diff saved to https://phabricator.wikimedia.org/P96951 and previous config saved to /var/cache/conftool/dbconfig/20261022-105205-ladsgroup.json
  • 12:51 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2003.codfw.wmnet
  • 12:51 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2002.codfw.wmnet
  • 12:51 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2002.codfw.wmnet
  • 12:48 dkertesz@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-upload_esams and A:cp - upgrade haproxy to 3.2.26-1 (T441435)
  • 12:44 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2002.codfw.wmnet
  • 12:42 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2156', diff saved to https://phabricator.wikimedia.org/P96949 and previous config saved to /var/cache/conftool/dbconfig/20261022-104203-ladsgroup.json
  • 12:41 slyngshede@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp5018.eqsin.wmnet
  • 12:41 slyngshede@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp5027.eqsin.wmnet
  • 12:39 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2002.codfw.wmnet
  • 12:39 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2001.codfw.wmnet
  • 12:39 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2001.codfw.wmnet
  • 12:35 dkertesz@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-text_codfw and A:cp - upgrade haproxy to 3.2.26-1 (T441435)
  • 12:34 fabfur@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp1105.eqiad.wmnet
  • 12:34 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-coord1003.eqiad.wmnet
  • 12:33 fabfur@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp1100.eqiad.wmnet
  • 12:32 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2001.codfw.wmnet
  • 12:32 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2156', diff saved to https://phabricator.wikimedia.org/P96948 and previous config saved to /var/cache/conftool/dbconfig/20261022-103201-ladsgroup.json
  • 12:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2001.codfw.wmnet
  • 12:31 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs-test2001.codfw.wmnet
  • 12:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs-test2001.codfw.wmnet
  • 12:29 slyngshede@cumin1004: START - Cookbook sre.cdn.roll-reboot rolling reboot on P{cp5018*} OR P{cp5019*} OR P{cp5020*} OR P{cp5021*} and A:cp
  • 12:28 slyngshede@cumin1004: START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-upload_eqsin and A:cp
  • 12:28 slyngshede@cumin1004: END (ERROR) - Cookbook sre.cdn.roll-reboot (exit_code=97) rolling reboot on A:cp-text_eqsin and A:cp
  • 12:28 slyngshede@cumin1004: START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-text_eqsin and A:cp
  • 12:26 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host an-coord1003.eqiad.wmnet
  • 12:25 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs-test2001.codfw.wmnet
  • 12:24 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs-test2001.codfw.wmnet
  • 12:24 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs2004.codfw.wmnet
  • 12:24 fabfur@cumin1004: START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-upload_eqiad
  • 12:24 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs2004.codfw.wmnet
  • 12:23 fabfur@cumin1004: START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-text_eqiad
  • 12:23 slyngshede@cumin1004: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P{cp4043*} OR P{cp4044*} OR P{cp4045*} OR P{cp4046*} and A:cp
  • 12:23 slyngshede@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp4046.ulsfo.wmnet
  • 12:22 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2156 (T441633)', diff saved to https://phabricator.wikimedia.org/P96947 and previous config saved to /var/cache/conftool/dbconfig/20261022-102159-ladsgroup.json
  • 12:19 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2004.codfw.wmnet
  • 12:17 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs2004.codfw.wmnet
  • 12:17 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs2004.codfw.wmnet
  • 12:16 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs2003.codfw.wmnet
  • 12:16 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs2003.codfw.wmnet
  • 12:15 elukey@cumin1004: START - Cookbook sre.hosts.reboot-single for host registry2004.codfw.wmnet
  • 12:15 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2005.codfw.wmnet
  • 12:14 dkertesz@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_esams and A:cp - upgrade haproxy to 3.2.26-1 (T441435)
  • 12:14 dkertesz@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_esams and A:cp - upgrade haproxy to 3.2.26-1 (T441435)
  • 12:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2156 (T441633)', diff saved to https://phabricator.wikimedia.org/P96945 and previous config saved to /var/cache/conftool/dbconfig/20261022-101329-ladsgroup.json
  • 12:13 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2156.codfw.wmnet with reason: Maintenance
  • 12:11 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs2003.codfw.wmnet
  • 12:11 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs2003.codfw.wmnet
  • 12:11 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs2002.codfw.wmnet
  • 12:11 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs2002.codfw.wmnet
  • 12:10 elukey@cumin1004: START - Cookbook sre.hosts.reboot-single for host registry2005.codfw.wmnet
  • 12:09 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1005.eqiad.wmnet
  • 12:08 dkertesz@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-upload_codfw and A:cp - upgrade haproxy to 3.2.26-1 (T441435)
  • 12:07 slyngshede@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp4045.ulsfo.wmnet
  • 12:05 elukey@cumin1004: START - Cookbook sre.hosts.reboot-single for host registry1005.eqiad.wmnet
  • 12:05 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1004.eqiad.wmnet
  • 12:04 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host pki-root1002.eqiad.wmnet
  • 12:04 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs2002.codfw.wmnet
  • 12:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs2002.codfw.wmnet
  • 12:03 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs2001.codfw.wmnet
  • 12:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs2001.codfw.wmnet
  • 12:00 elukey@cumin1004: START - Cookbook sre.hosts.reboot-single for host registry1004.eqiad.wmnet
  • 12:00 btullis@dns1004: END - running authdns-update
  • 11:59 elukey@cumin1004: START - Cookbook sre.hosts.reboot-single for host pki-root1002.eqiad.wmnet
  • 11:57 btullis@dns1004: START - running authdns-update
  • 11:56 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs2001.codfw.wmnet
  • 11:56 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs2001.codfw.wmnet
  • 11:56 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:dse-k8s-worker-codfw
  • 11:50 slyngshede@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp4044.ulsfo.wmnet
  • 11:48 dkertesz@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-drmrs and A:cp - upgrade haproxy to 3.2.26-1 (T441435)
  • 11:47 slyngshede@cumin1004: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P{cp5017*} and A:cp
  • 11:47 slyngshede@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp5017.eqsin.wmnet
  • 11:42 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host db1208.eqiad.wmnet
  • 11:40 dkertesz@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_codfw and A:cp - upgrade haproxy to 3.2.26-1 (T441435)
  • 11:39 dkertesz@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_codfw and A:cp - upgrade haproxy to 3.2.26-1 (T441435)
  • 11:35 slyngshede@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp4043.ulsfo.wmnet
  • 11:34 slyngshede@cumin1004: START - Cookbook sre.cdn.roll-reboot rolling reboot on P{cp5017*} and A:cp
  • 11:28 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host db1208.eqiad.wmnet
  • 11:25 slyngshede@cumin1004: START - Cookbook sre.cdn.roll-reboot rolling reboot on P{cp4043*} OR P{cp4044*} OR P{cp4045*} OR P{cp4046*} and A:cp
  • 11:23 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-test-presto1001.eqiad.wmnet
  • 11:23 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host zookeeper-test1002.eqiad.wmnet
  • 11:23 slyngshede@cumin1004: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P{cp4042*} and A:cp
  • 11:23 slyngshede@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp4042.ulsfo.wmnet
  • 11:22 slyngshede@cumin1004: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P{cp4052*} and A:cp
  • 11:22 slyngshede@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp4052.ulsfo.wmnet
  • 11:19 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host an-test-presto1001.eqiad.wmnet
  • 11:19 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host zookeeper-test1002.eqiad.wmnet
  • 11:15 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-test-druid1001.eqiad.wmnet
  • 11:12 slyngshede@cumin1004: START - Cookbook sre.cdn.roll-reboot rolling reboot on P{cp4052*} and A:cp
  • 11:12 slyngshede@cumin1004: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P{cp4051*} and A:cp
  • 11:12 slyngshede@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp4051.ulsfo.wmnet
  • 11:11 slyngshede@cumin1004: START - Cookbook sre.cdn.roll-reboot rolling reboot on P{cp4042*} and A:cp
  • 11:10 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host an-test-druid1001.eqiad.wmnet
  • 11:09 slyngshede@cumin1004: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P{cp4041*} and A:cp
  • 11:09 slyngshede@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp4041.ulsfo.wmnet
  • 11:00 slyngshede@cumin1004: START - Cookbook sre.cdn.roll-reboot rolling reboot on P{cp4051*} and A:cp
  • 11:00 btullis@cumin1004: END (PASS) - Cookbook sre.hadoop.reboot-workers (exit_code=0) for Hadoop test cluster
  • 10:59 slyngshede@cumin1004: START - Cookbook sre.cdn.roll-reboot rolling reboot on P{cp4041*} and A:cp
  • 10:23 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-test-master1004.eqiad.wmnet
  • 10:17 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host an-test-master1004.eqiad.wmnet
  • 10:16 btullis@cumin1004: START - Cookbook sre.hadoop.reboot-workers for Hadoop test cluster
  • 10:05 dkertesz@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-drmrs and A:cp - upgrade haproxy to 3.2.26-1 (T441435)
  • 09:21 arthurtaylor@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply
  • 09:21 arthurtaylor@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply
  • 09:19 arthurtaylor@deploy1003: helmfile [staging] DONE helmfile.d/services/wikidata-query-gui: apply
  • 09:19 arthurtaylor@deploy1003: helmfile [staging] START helmfile.d/services/wikidata-query-gui: apply
  • 09:19 slyngshede@cumin1004: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P{cp4040*} and A:cp
  • 09:19 slyngshede@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp4040.ulsfo.wmnet
  • 09:19 slyngshede@cumin1004: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P{cp4050*} and A:cp
  • 09:19 slyngshede@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp4050.ulsfo.wmnet
  • 09:10 slyngshede@cumin1004: START - Cookbook sre.cdn.roll-reboot rolling reboot on P{cp4040*} and A:cp
  • 09:10 slyngshede@cumin1004: START - Cookbook sre.cdn.roll-reboot rolling reboot on P{cp4050*} and A:cp
  • 09:06 slyngshede@cumin1004: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P{cp4039*} and A:cp
  • 09:06 slyngshede@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp4039.ulsfo.wmnet
  • 09:03 slyngshede@cumin1004: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P{cp4049*} and A:cp
  • 09:03 slyngshede@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp4049.ulsfo.wmnet
  • 08:54 slyngshede@cumin1004: START - Cookbook sre.cdn.roll-reboot rolling reboot on P{cp4049*} and A:cp
  • 08:54 slyngshede@cumin1004: START - Cookbook sre.cdn.roll-reboot rolling reboot on P{cp4039*} and A:cp
  • 08:49 slyngshede@cumin1004: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P{cp4038*} and A:cp
  • 08:49 slyngshede@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp4038.ulsfo.wmnet
  • 08:48 slyngshede@cumin1004: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P{cp4048*} and A:cp
  • 08:48 slyngshede@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp4048.ulsfo.wmnet
  • 08:39 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2239.codfw.wmnet with OS trixie
  • 08:38 slyngshede@cumin1004: START - Cookbook sre.cdn.roll-reboot rolling reboot on P{cp4038*} and A:cp
  • 08:38 slyngshede@cumin1004: START - Cookbook sre.cdn.roll-reboot rolling reboot on P{cp4048*} and A:cp
  • 08:36 slyngshede@cumin1004: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P{cp4037*} and A:cp
  • 08:36 slyngshede@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp4037.ulsfo.wmnet
  • 08:36 slyngshede@cumin1004: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P{cp4047*} and A:cp
  • 08:36 slyngshede@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp4047.ulsfo.wmnet
  • 08:26 slyngshede@cumin1004: START - Cookbook sre.cdn.roll-reboot rolling reboot on P{cp4047*} and A:cp
  • 08:24 slyngshede@cumin1004: START - Cookbook sre.cdn.roll-reboot rolling reboot on P{cp4037*} and A:cp
  • 08:07 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2239.codfw.wmnet with reason: host reimage
  • 08:04 marostegui@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on db2239.codfw.wmnet with reason: host reimage
  • 07:48 marostegui@cumin1004: START - Cookbook sre.hosts.reimage for host db2239.codfw.wmnet with OS trixie
  • 07:45 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db2239.codfw.wmnet with reason: Upgrade
  • 07:33 marostegui@cumin1004: dbctl commit (dc=all): 'Remove db1175 T441800', diff saved to https://phabricator.wikimedia.org/P96941 and previous config saved to /var/cache/conftool/dbconfig/20261022-053348-marostegui.json
  • 07:15 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on dbproxy1026.eqiad.wmnet with reason: Restart
  • 07:11 marostegui@dns1004: END - running authdns-update
  • 07:08 marostegui@dns1004: START - running authdns-update
  • 02:38 dzahn@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 6 hosts with reason: temporary disabled
  • 02:00 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
  • 02:00 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
  • 02:00 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
  • 02:00 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: sync
  • 00:35 herron@cumin1004: END (FAIL) - Cookbook sre.opensearch.roll-restart-reboot (exit_code=99) rolling reboot on A:logstash-eqiad

2026-10-21

  • 23:24 jasmine@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host parsoidtest1001.eqiad.wmnet with OS trixie
  • 23:14 herron@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host grafana2001.codfw.wmnet
  • 23:10 herron@cumin1004: START - Cookbook sre.hosts.reboot-single for host grafana2001.codfw.wmnet
  • 23:07 herron@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host grafana1002.eqiad.wmnet
  • 23:03 herron@cumin1004: START - Cookbook sre.hosts.reboot-single for host grafana1002.eqiad.wmnet
  • 23:01 herron@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host centrallog2002.codfw.wmnet
  • 22:54 herron@cumin1004: START - Cookbook sre.hosts.reboot-single for host centrallog2002.codfw.wmnet
  • 22:51 herron@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host centrallog1002.eqiad.wmnet
  • 22:44 herron@cumin1004: START - Cookbook sre.hosts.reboot-single for host centrallog1002.eqiad.wmnet
  • 22:42 herron@cumin1004: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host alert2002.wikimedia.org
  • 22:42 herron@cumin1004: START - Cookbook sre.hosts.reboot-single for host alert2002.wikimedia.org
  • 22:40 dzahn@cumin1004: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: security release
  • 22:39 herron@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host alert1002.wikimedia.org
  • 22:31 herron@cumin1004: START - Cookbook sre.hosts.reboot-single for host alert1002.wikimedia.org
  • 22:29 mutante: doc1004 - systemctl start wmf_auto_restart_envoyproxy.service - clear monitoring alerts due to previous envoy downtime
  • 22:14 jasmine@cumin2003: START - Cookbook sre.hosts.reimage for host parsoidtest1001.eqiad.wmnet with OS trixie
  • 22:14 brett@cumin2003: END (PASS) - Cookbook sre.cdn.roll-upgrade-ats (exit_code=0) Rolling upgrade of ATS on P{cp7016.magru.wmnet} and A:cp - 9.2.16 upgrade (T441945)
  • 22:09 dzahn@cumin1004: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: security release
  • 22:09 brett@cumin2003: START - Cookbook sre.cdn.roll-upgrade-ats Rolling upgrade of ATS on P{cp7016.magru.wmnet} and A:cp - 9.2.16 upgrade (T441945)
  • 22:09 brett@cumin2003: END (PASS) - Cookbook sre.cdn.roll-upgrade-ats (exit_code=0) Rolling upgrade of ATS on P{cp7001.magru.wmnet} and A:cp - 9.2.16 upgrade (T441945)
  • 22:03 brett@cumin2003: START - Cookbook sre.cdn.roll-upgrade-ats Rolling upgrade of ATS on P{cp7001.magru.wmnet} and A:cp - 9.2.16 upgrade (T441945)
  • 21:58 sukhe@cumin1004: END (PASS) - Cookbook sre.dns.roll-reboot (exit_code=0) rolling reboot on P{dns6002* or dns5* or dns4004*} and (A:dnsbox)
  • 21:58 sukhe@cumin1004: cookbooks.sre.dns.roll-reboot finished rebooting dns6002.wikimedia.org
  • 21:57 brett: import trafficserver 9.2.16~deb13+wmf1 into trixie-wikimedia - T441945
  • 21:57 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply
  • 21:56 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply
  • 21:55 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply
  • 21:48 sukhe@cumin1004: cookbooks.sre.dns.roll-reboot begin reboot of dns6002.wikimedia.org
  • 21:39 bking@cumin2003: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_eqiad: apply kernel updates - bking@cumin2003 - T441838
  • 21:36 brett@cumin2003: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-text_eqsin - 3.2.26 upgrade (T441435)
  • 21:33 sukhe@cumin1004: cookbooks.sre.dns.roll-reboot finished rebooting dns5004.wikimedia.org
  • 21:31 dzahn@cumin1004: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab2002.wikimedia.org with reason: security release
  • 21:21 brett@cumin2003: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-text_ulsfo - 3.2.26 upgrade (T441435)
  • 21:21 dzahn@cumin1004: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab2002.wikimedia.org with reason: security release
  • 21:20 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply
  • 21:19 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply
  • 21:17 sukhe@cumin1004: cookbooks.sre.dns.roll-reboot begin reboot of dns5004.wikimedia.org
  • 21:14 dzahn@cumin1004: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1003.wikimedia.org with reason: security release
  • 21:12 dancy@deploy1003: Locking from deployment [ALL REPOSITORIES]: Deployments locked for the week
  • 21:05 dzahn@cumin1004: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: security release
  • 21:04 brett@cumin2003: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-upload_eqsin - 3.2.26 upgrade (T441435)
  • 21:02 sukhe@cumin1004: cookbooks.sre.dns.roll-reboot finished rebooting dns5003.wikimedia.org
  • 21:00 dzahn@cumin1004: END (FAIL) - Cookbook sre.gitlab.upgrade (exit_code=99) on GitLab host gitlab1003.wikimedia.org with reason: security release
  • 21:00 dzahn@cumin1004: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: security release
  • 20:56 brett@cumin2003: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-upload_magru - 3.2.26 upgrade (T441435)
  • 20:56 brett@cumin2003: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-upload_ulsfo - 3.2.26 upgrade (T441435)
  • 20:54 bking@cumin2003: START - Cookbook sre.elasticsearch.rolling-operation Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_eqiad: apply kernel updates - bking@cumin2003 - T441838
  • 20:51 sukhe@cumin1004: cookbooks.sre.dns.roll-reboot begin reboot of dns5003.wikimedia.org
  • 20:48 inflatador: bking@an-launcher1003 `systemctl restart mediawiki-private-raw-actor-drop-weekly.service` attempt to fix failed HDFS reporting job
  • 20:45 bking@cumin2003: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_eqiad: apply kernel updates - bking@cumin2003 - T441838
  • 20:36 sukhe@cumin1004: cookbooks.sre.dns.roll-reboot finished rebooting dns4004.wikimedia.org
  • 20:27 dzahn@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on contint2002.wikimedia.org with reason: zuul currently down - redirects to jenkins
  • 20:27 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=dns4003.wikimedia.org
  • 20:27 dzahn@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on contint1002.wikimedia.org with reason: zuul currently down - redirects to jenkins
  • 20:27 sukhe@cumin1004: cookbooks.sre.dns.roll-reboot begin reboot of dns4004.wikimedia.org
  • 20:27 sukhe@cumin1004: START - Cookbook sre.dns.roll-reboot rolling reboot on P{dns6002* or dns5* or dns4004*} and (A:dnsbox)
  • 20:26 sukhe@cumin1004: END (ERROR) - Cookbook sre.dns.roll-reboot (exit_code=97) rolling reboot on P{dns6002* or dns5* or dns4*} and (A:dnsbox)
  • 20:26 sukhe@cumin1004: cookbooks.sre.dns.roll-reboot begin reboot of dns4003.wikimedia.org
  • 20:26 sukhe@cumin1004: START - Cookbook sre.dns.roll-reboot rolling reboot on P{dns6002* or dns5* or dns4*} and (A:dnsbox)
  • 20:25 mutante: contint1002/2002 - edited apache config change - redirect zuul to jenkins
  • 20:25 brett@cumin2003: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_ulsfo - 3.2.26 upgrade (T441435)
  • 20:25 brett@cumin2003: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_ulsfo - 3.2.26 upgrade (T441435)
  • 20:22 brett@cumin2003: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_eqsin - 3.2.26 upgrade (T441435)
  • 20:22 brett@cumin2003: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_eqsin - 3.2.26 upgrade (T441435)
  • 20:21 brett@cumin2003: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_magru - 3.2.26 upgrade (T441435)
  • 19:23 bking@cumin2003: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.REBOOT (1 nodes at a time) for ElasticSearch cluster cloudelastic: apply kernel updates - bking@cumin2003 - T441838
  • 18:57 bking@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on 6 hosts with reason: applying kernel updates
  • 18:55 herron@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host o11ytest2001.codfw.wmnet
  • 18:52 bking@cumin2003: START - Cookbook sre.elasticsearch.rolling-operation Operation.REBOOT (1 nodes at a time) for ElasticSearch cluster cloudelastic: apply kernel updates - bking@cumin2003 - T441838
  • 18:49 herron@cumin1004: START - Cookbook sre.hosts.reboot-single for host o11ytest2001.codfw.wmnet
  • 18:44 herron@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host o11ytest1001.eqiad.wmnet
  • 18:41 bking@cumin2003: START - Cookbook sre.elasticsearch.rolling-operation Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_eqiad: apply kernel updates - bking@cumin2003 - T441838
  • 18:38 herron@cumin1004: START - Cookbook sre.hosts.reboot-single for host o11ytest1001.eqiad.wmnet
  • 18:33 herron@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mwlog2003.codfw.wmnet
  • 18:26 herron@cumin1004: START - Cookbook sre.hosts.reboot-single for host mwlog2003.codfw.wmnet
  • 18:21 herron@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mwlog1003.eqiad.wmnet
  • 18:16 kamila@deploy1003: Finished scap deploy-service shellbox: update image (duration: 02m 29s)
  • 18:15 kamila@deploy1003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
  • 18:14 kamila@deploy1003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] codfw Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
  • 18:14 kamila@deploy1003: scap deploy-service shellbox: [STAGING/DEFAULT] staging-eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
  • 18:14 kamila@deploy1003: Started scap deploy-service shellbox: update image
  • 18:13 herron@cumin1004: START - Cookbook sre.hosts.reboot-single for host mwlog1003.eqiad.wmnet
  • 18:08 herron@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host kafkamon2003.codfw.wmnet
  • 18:07 herron@cumin1004: START - Cookbook sre.hosts.reboot-single for host kafkamon2003.codfw.wmnet
  • 18:02 herron@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host kafkamon1003.eqiad.wmnet
  • 18:01 herron@cumin1004: START - Cookbook sre.hosts.reboot-single for host kafkamon1003.eqiad.wmnet
  • 17:46 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-misc: apply
  • 17:46 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-misc: apply
  • 17:42 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/mw-misc: apply
  • 17:42 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-misc: apply
  • 17:41 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: sync
  • 17:40 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: sync
  • 17:40 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 17:40 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 17:36 herron@cumin1004: START - Cookbook sre.opensearch.roll-restart-reboot rolling reboot on A:logstash-eqiad
  • 17:34 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-wikifunctions: apply
  • 17:34 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-wikifunctions: apply
  • 17:33 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-pretrain: apply
  • 17:33 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-pretrain: apply
  • 17:30 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/mw-pretrain: apply
  • 17:29 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-pretrain: apply
  • 17:29 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-videoscaler: apply
  • 17:29 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/mw-videoscaler: apply
  • 17:29 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-videoscaler: apply
  • 17:29 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-videoscaler: apply
  • 17:28 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-jobrunner: apply
  • 17:28 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-jobrunner: apply
  • 17:28 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-jobrunner: apply
  • 17:27 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/mw-jobrunner: apply
  • 17:27 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-wikifunctions: apply
  • 17:27 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/mw-wikifunctions: apply
  • 17:26 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/mw-wikifunctions: apply
  • 17:26 mvernon@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host thanos-be2009.codfw.wmnet
  • 17:25 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
  • 17:25 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: sync
  • 17:25 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: sync
  • 17:25 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: sync
  • 17:25 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: sync
  • 17:25 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
  • 17:25 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply
  • 17:25 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
  • 17:24 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: sync
  • 17:24 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: sync
  • 17:24 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply
  • 17:24 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply
  • 17:23 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
  • 17:21 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
  • 17:21 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-web: apply
  • 17:20 sukhe@cumin1004: END (FAIL) - Cookbook sre.dns.admin (exit_code=99) DNS admin: depool codfw [reason: no reason specified, no task ID specified]
  • 17:20 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply
  • 17:20 sukhe@cumin1004: START - Cookbook sre.dns.admin DNS admin: depool codfw [reason: no reason specified, no task ID specified]
  • 17:19 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
  • 17:18 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
  • 17:18 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-web: apply
  • 17:17 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply
  • 17:17 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
  • 17:17 mvernon@cumin1004: START - Cookbook sre.hosts.reboot-single for host thanos-be2009.codfw.wmnet
  • 17:17 mvernon@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host thanos-be1009.eqiad.wmnet
  • 17:16 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply
  • 17:16 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
  • 17:15 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 17:15 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-ext: apply
  • 17:14 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 17:13 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 17:12 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/mw-web: apply
  • 17:11 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 17:10 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply
  • 17:10 dancy@deploy1003: sync-world aborted: (no justification provided) (duration: 16m 09s)
  • 17:09 mvernon@cumin1004: START - Cookbook sre.hosts.reboot-single for host thanos-be1009.eqiad.wmnet
  • 17:09 mvernon@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host thanos-be2008.codfw.wmnet
  • 17:09 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-web: apply
  • 17:02 mvernon@cumin1004: START - Cookbook sre.hosts.reboot-single for host thanos-be2008.codfw.wmnet
  • 17:02 mvernon@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host thanos-be1008.eqiad.wmnet
  • 16:54 dancy@deploy1003: Started scap sync-world: (no justification provided)
  • 16:54 mvernon@cumin1004: START - Cookbook sre.hosts.reboot-single for host thanos-be1008.eqiad.wmnet
  • 16:54 mvernon@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host thanos-be2007.codfw.wmnet
  • 16:54 hashar@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: Deployments locked for the week (duration: 307m 09s)
  • 16:54 dancy@deploy1003: Forcefully removing global lock: Deployments locked for the week
  • 16:47 mvernon@cumin1004: START - Cookbook sre.hosts.reboot-single for host thanos-be2007.codfw.wmnet
  • 16:47 mvernon@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host thanos-be1007.eqiad.wmnet
  • 16:38 mvernon@cumin1004: START - Cookbook sre.hosts.reboot-single for host thanos-be1007.eqiad.wmnet
  • 16:38 mvernon@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host thanos-be2006.codfw.wmnet
  • 16:31 mvernon@cumin1004: START - Cookbook sre.hosts.reboot-single for host thanos-be2006.codfw.wmnet
  • 16:31 mvernon@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host thanos-be1006.eqiad.wmnet
  • 16:24 mvernon@cumin1004: START - Cookbook sre.hosts.reboot-single for host thanos-be1006.eqiad.wmnet
  • 16:24 mvernon@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host thanos-be2005.codfw.wmnet
  • 16:16 mvernon@cumin1004: START - Cookbook sre.hosts.reboot-single for host thanos-be2005.codfw.wmnet
  • 16:16 mvernon@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host thanos-be1005.eqiad.wmnet
  • 16:07 mvernon@cumin1004: START - Cookbook sre.hosts.reboot-single for host thanos-be1005.eqiad.wmnet
  • 14:58 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on parsoidtest1001.eqiad.wmnet with reason: maint
  • 14:54 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host dbstore1009.eqiad.wmnet
  • 14:36 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-master1003.eqiad.wmnet
  • 14:32 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dbstore1009.eqiad.wmnet
  • 14:27 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dbstore1008.eqiad.wmnet
  • 14:09 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host an-master1003.eqiad.wmnet
  • 14:08 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dbstore1008.eqiad.wmnet
  • 14:00 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
  • 14:00 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: sync
  • 13:43 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host dbstore1007.eqiad.wmnet
  • 13:20 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dbstore1007.eqiad.wmnet
  • 13:07 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply
  • 13:06 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply
  • 13:06 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply
  • 13:05 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply
  • 12:46 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
  • 12:46 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
  • 12:45 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
  • 12:44 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
  • 12:44 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
  • 12:43 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
  • 12:42 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
  • 12:41 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:ml-serve-worker-codfw
  • 12:40 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve2011.codfw.wmnet
  • 12:40 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve2011.codfw.wmnet
  • 12:36 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve2011.codfw.wmnet
  • 12:31 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve2011.codfw.wmnet
  • 12:31 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve2010.codfw.wmnet
  • 12:31 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve2010.codfw.wmnet
  • 12:26 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve2010.codfw.wmnet
  • 12:21 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve2010.codfw.wmnet
  • 12:21 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve2009.codfw.wmnet
  • 12:21 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve2009.codfw.wmnet
  • 12:16 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve2009.codfw.wmnet
  • 12:12 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-redacteddb1001.eqiad.wmnet
  • 12:10 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve2009.codfw.wmnet
  • 12:10 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve2008.codfw.wmnet
  • 12:10 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve2008.codfw.wmnet
  • 12:05 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve2008.codfw.wmnet
  • 11:57 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host an-redacteddb1001.eqiad.wmnet
  • 11:55 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve2008.codfw.wmnet
  • 11:55 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve2007.codfw.wmnet
  • 11:55 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve2007.codfw.wmnet
  • 11:50 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve2007.codfw.wmnet
  • 11:46 hashar@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: Deployments locked for the week (duration: 00m 00s)
  • 11:46 hashar@deploy1003: Locking from deployment [ALL REPOSITORIES]: Deployments locked for the week
  • 11:45 hashar@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: Deployments locked for the week (duration: 00m 07s)
  • 11:45 hashar@deploy1003: Locking from deployment [ALL REPOSITORIES]: Deployments locked for the week
  • 11:45 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve2007.codfw.wmnet
  • 11:45 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve2006.codfw.wmnet
  • 11:45 hashar@deploy1003: Finished deploy [gerrit/gerrit@6865599]: Add banner about emergency maintenance (duration: 00m 16s)
  • 11:45 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve2006.codfw.wmnet
  • 11:44 hashar@deploy1003: Started deploy [gerrit/gerrit@6865599]: Add banner about emergency maintenance
  • 11:44 dancy@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: Deployments locked for the week (duration: 532m 38s)
  • 11:44 hashar@deploy1003: Forcefully removing global lock: Emergency deploy for gerrit/gerrit
  • 11:40 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve2006.codfw.wmnet
  • 11:36 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-worker1219.eqiad.wmnet
  • 11:34 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve2006.codfw.wmnet
  • 11:34 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve2005.codfw.wmnet
  • 11:34 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve2005.codfw.wmnet
  • 11:30 arnaudb@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on gerrit2002.wikimedia.org with reason: T440177 offline reindex test
  • 11:29 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve2005.codfw.wmnet
  • 11:29 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host an-worker1219.eqiad.wmnet
  • 11:24 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-worker1213.eqiad.wmnet
  • 11:24 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve2005.codfw.wmnet
  • 11:24 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve2004.codfw.wmnet
  • 11:24 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve2004.codfw.wmnet
  • 11:22 jiji@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool mw-web-ro in eqiad: T440896
  • 11:20 effie: repooled mw-web on eqiad
  • 11:17 jiji@cumin1004: START - Cookbook sre.discovery.service-route pool mw-web-ro in eqiad: T440896
  • 11:17 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve2004.codfw.wmnet
  • 11:17 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host an-worker1213.eqiad.wmnet
  • 11:13 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-worker1178.eqiad.wmnet
  • 11:11 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es1039.eqiad.wmnet with reason: Crashing
  • 11:07 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve2004.codfw.wmnet
  • 11:07 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve2003.codfw.wmnet
  • 11:07 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve2003.codfw.wmnet
  • 11:03 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host an-worker1178.eqiad.wmnet
  • 11:02 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve2003.codfw.wmnet
  • 10:57 jiji@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool mw-api-ext-ro in eqiad: T440896
  • 10:57 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve2003.codfw.wmnet
  • 10:56 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve2002.codfw.wmnet
  • 10:56 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve2002.codfw.wmnet
  • 10:54 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-worker1151.eqiad.wmnet
  • 10:52 jiji@cumin1004: START - Cookbook sre.discovery.service-route pool mw-api-ext-ro in eqiad: T440896
  • 10:51 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve2002.codfw.wmnet
  • 10:46 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host an-worker1151.eqiad.wmnet
  • 10:46 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve2002.codfw.wmnet
  • 10:46 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve2001.codfw.wmnet
  • 10:46 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve2001.codfw.wmnet
  • 10:40 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve2001.codfw.wmnet
  • 10:35 jiji@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool mw-api-int-ro in eqiad: T440896
  • 10:35 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve2001.codfw.wmnet
  • 10:35 klausman@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:ml-serve-worker-codfw
  • 10:30 jiji@cumin1004: START - Cookbook sre.discovery.service-route pool mw-api-int-ro in eqiad: T440896
  • 10:20 arnaudb@dns1006: END - running authdns-update
  • 10:17 arnaudb@dns1006: START - running authdns-update
  • 10:09 arnaudb@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es1039.eqiad.wmnet with reason: host p.aged
  • 10:05 arnaudb@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'.
  • 07:55 TimStarling: on phab1005 wrote a little python server to serve PHP security error pages and ran it
  • 05:40 rzl@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-video: apply
  • 05:39 rzl@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-video: apply
  • 05:39 rzl@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply
  • 05:39 rzl@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-video: apply
  • 05:38 rzl@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-video: apply
  • 05:38 rzl@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-video: apply
  • 05:38 rzl@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: apply
  • 05:38 rzl@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: apply
  • 05:36 rzl@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-timeline: apply
  • 05:36 rzl@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-timeline: apply
  • 05:35 rzl@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: apply
  • 05:35 rzl@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: apply
  • 05:34 rzl@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply
  • 05:34 rzl@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply
  • 05:34 rzl@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-syntaxhighlight: apply
  • 05:34 rzl@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-syntaxhighlight: apply
  • 05:34 rzl@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-syntaxhighlight: apply
  • 05:33 rzl@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-syntaxhighlight: apply
  • 05:33 rzl@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-media: apply
  • 05:33 rzl@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-media: apply
  • 05:33 rzl@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-media: apply
  • 05:33 rzl@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-media: apply
  • 05:33 rzl@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-media: apply
  • 05:32 rzl@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-media: apply
  • 05:32 rzl@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-constraints: apply
  • 05:32 rzl@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-constraints: apply
  • 05:32 rzl@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-constraints: apply
  • 05:32 rzl@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-constraints: apply
  • 05:31 rzl@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-constraints: apply
  • 05:31 rzl@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-constraints: apply
  • 05:16 rzl@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox: apply
  • 05:15 rzl@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox: apply
  • 05:15 rzl@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox: apply
  • 05:14 rzl@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox: apply
  • 05:10 rzl@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: apply
  • 05:10 rzl@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: apply
  • 04:40 rzl@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: apply
  • 04:30 rzl@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: apply
  • 04:09 jasmine@cumin2003: dbctl commit (dc=all): 'Depool es1039', diff saved to and previous config saved to /var/cache/conftool/dbconfig/20261021-020908-jasmine.json
  • 04:03 jasmine@cumin2003: END (FAIL) - Cookbook sre.mysql.depool (exit_code=99) depool es1039.eqiad.wmnet: Depooling replica
  • 03:45 jasmine@cumin2003: START - Cookbook sre.mysql.depool depool es1039.eqiad.wmnet: Depooling replica
  • 03:06 herron@cumin1004: END (FAIL) - Cookbook sre.opensearch.roll-restart-reboot (exit_code=99) rolling reboot on A:logstash-eqiad
  • 02:52 dancy@deploy1003: Locking from deployment [ALL REPOSITORIES]: Deployments locked for the week
  • 02:51 dancy@deploy1003: Finished scap sync-world: Testing (duration: 25m 52s)
  • 02:39 dancy@deploy1003: dancy: Continuing with deployment
  • 02:37 dancy@deploy1003: dancy: Testing synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 02:35 bblack@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on phab1005.eqiad.wmnet with reason: php
  • 02:26 dancy@deploy1003: Started scap sync-world: Testing
  • 02:25 dancy@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: Deployments locked for the week (duration: 00m 58s)
  • 02:25 dancy@deploy1003: Forcefully removing global lock: Deployments locked for the week
  • 02:24 dancy@deploy1003: Locking from deployment [ALL REPOSITORIES]: Deployments locked for the week
  • 02:24 dancy@deploy1003: Finished scap build-images: (no justification provided) (duration: 11m 59s)
  • 02:17 bking@cumin2003: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_codfw: apply kernel updates - bking@cumin2003 - T441838
  • 02:12 dancy@deploy1003: Started scap build-images: (no justification provided)
  • 02:07 dancy@deploy1003: Started scap build-images: (no justification provided)
  • 02:07 thcipriani@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: Deployments locked for the week (duration: 33m 46s)
  • 02:07 dancy@deploy1003: Forcefully removing global lock: Deployments locked for the week
  • 02:00 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
  • 02:00 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: sync
  • 01:33 thcipriani@deploy1003: Locking from deployment [ALL REPOSITORIES]: Deployments locked for the week
  • 01:29 thcipriani@deploy1003: Finished scap sync-world: Backport for PhpVersion: Show only the major and minor PHP version (1.47.0-wmf.24, 1.48.0-wmf.1) (duration: 14m 13s)
  • 01:23 thcipriani@deploy1003: thcipriani: Continuing with deployment
  • 01:19 thcipriani@deploy1003: thcipriani: Backport for PhpVersion: Show only the major and minor PHP version (1.47.0-wmf.24, 1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 01:15 thcipriani@deploy1003: Started scap sync-world: Backport for PhpVersion: Show only the major and minor PHP version (1.47.0-wmf.24, 1.48.0-wmf.1)
  • 01:12 thcipriani@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: Deployments locked for the week (duration: 107m 04s)
  • 01:11 thcipriani@deploy1003: Forcefully removing global lock: Pausing deploy lock for quick backport
  • 00:10 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.reboot (exit_code=0)
  • 00:02 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs1027.eqiad.wmnet

2026-10-20

  • 23:55 bking@cumin2003: START - Cookbook sre.hosts.reboot-single for host wdqs1027.eqiad.wmnet
  • 23:51 bking@cumin2003: START - Cookbook sre.wdqs.reboot
  • 23:49 bking@cumin2003: END (FAIL) - Cookbook sre.wdqs.reboot (exit_code=99)
  • 23:49 bking@cumin2003: START - Cookbook sre.wdqs.reboot
  • 23:41 bking@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on cloudelastic1008.eqiad.wmnet with reason: backplane error
  • 23:39 bking@cumin2003: END (ERROR) - Cookbook sre.elasticsearch.rolling-operation (exit_code=97) Operation.REBOOT (1 nodes at a time) for ElasticSearch cluster cloudelastic: apply kernel updates - bking@cumin2003 - T441838
  • 23:24 thcipriani@deploy1003: Locking from deployment [ALL REPOSITORIES]: Deployments locked for the week
  • 23:19 thcipriani@deploy1003: thcipriani: Backport for PhpVersion: Show only the major and minor PHP version (1.47.0-wmf.24, 1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 23:15 thcipriani@deploy1003: Started scap sync-world: Backport for PhpVersion: Show only the major and minor PHP version (1.47.0-wmf.24, 1.48.0-wmf.1)
  • 23:12 thcipriani@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: Deployments locked for the week (duration: 107m 04s)
  • 23:11 thcipriani@deploy1003: Forcefully removing global lock: Pausing deploy lock for quick backport
  • 22:10 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.reboot (exit_code=0)
  • 22:02 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs1027.eqiad.wmnet
  • 21:55 bking@cumin2003: START - Cookbook sre.hosts.reboot-single for host wdqs1027.eqiad.wmnet
  • 21:51 bking@cumin2003: START - Cookbook sre.wdqs.reboot
  • 21:49 bking@cumin2003: END (FAIL) - Cookbook sre.wdqs.reboot (exit_code=99)
  • 21:49 bking@cumin2003: START - Cookbook sre.wdqs.reboot
  • 21:41 bking@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on cloudelastic1008.eqiad.wmnet with reason: backplane error
  • 21:39 bking@cumin2003: END (ERROR) - Cookbook sre.elasticsearch.rolling-operation (exit_code=97) Operation.REBOOT (1 nodes at a time) for ElasticSearch cluster cloudelastic: apply kernel updates - bking@cumin2003 - T441838
  • 21:24 thcipriani@deploy1003: Locking from deployment [ALL REPOSITORIES]: Deployments locked for the week
  • 21:14 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.reboot (exit_code=0)
  • 21:05 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1009.eqiad.wmnet with reason: Maintenance
  • 21:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1287 (T441633)', diff saved to https://phabricator.wikimedia.org/P96937 and previous config saved to /var/cache/conftool/dbconfig/20261020-210526-ladsgroup.json
  • 20:55 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1287', diff saved to https://phabricator.wikimedia.org/P96935 and previous config saved to /var/cache/conftool/dbconfig/20261020-205524-ladsgroup.json
  • 20:53 bking@cumin2003: START - Cookbook sre.wdqs.reboot
  • 20:53 bking@cumin2003: END (FAIL) - Cookbook sre.wdqs.reboot (exit_code=99)
  • 20:53 bking@cumin2003: START - Cookbook sre.wdqs.reboot
  • 20:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1287', diff saved to https://phabricator.wikimedia.org/P96934 and previous config saved to /var/cache/conftool/dbconfig/20261020-204522-ladsgroup.json
  • 20:39 arlolra@deploy1003: Finished scap sync-world: Backport for Undeploy Parsoid readviews on hewikisource (T441087) (duration: 10m 37s)
  • 20:38 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs2025.codfw.wmnet
  • 20:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1287 (T441633)', diff saved to https://phabricator.wikimedia.org/P96933 and previous config saved to /var/cache/conftool/dbconfig/20261020-203520-ladsgroup.json
  • 20:34 arlolra@deploy1003: ssastry, arlolra: Continuing with deployment
  • 20:33 arlolra@deploy1003: ssastry, arlolra: Backport for Undeploy Parsoid readviews on hewikisource (T441087) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 20:30 bking@cumin2003: START - Cookbook sre.hosts.reboot-single for host wdqs2025.codfw.wmnet
  • 20:30 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1287 (T441633)', diff saved to https://phabricator.wikimedia.org/P96932 and previous config saved to /var/cache/conftool/dbconfig/20261020-203016-ladsgroup.json
  • 20:30 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1287.eqiad.wmnet with reason: Maintenance
  • 20:29 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1282 (T441633)', diff saved to https://phabricator.wikimedia.org/P96931 and previous config saved to /var/cache/conftool/dbconfig/20261020-202957-ladsgroup.json
  • 20:29 arlolra@deploy1003: Started scap sync-world: Backport for Undeploy Parsoid readviews on hewikisource (T441087)
  • 20:25 arlolra@deploy1003: Finished scap sync-world: Backport for Add "proofread-index" to LINTABLE_CONTENT_MODELS (T441874) (1.48.0-wmf.1, 1.47.0-wmf.24) (duration: 18m 59s)
  • 20:19 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1282', diff saved to https://phabricator.wikimedia.org/P96930 and previous config saved to /var/cache/conftool/dbconfig/20261020-201954-ladsgroup.json
  • 20:11 arlolra@deploy1003: arlolra: Continuing with deployment
  • 20:11 bking@cumin2003: START - Cookbook sre.elasticsearch.rolling-operation Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_codfw: apply kernel updates - bking@cumin2003 - T441838
  • 20:11 bking@cumin2003: END (ERROR) - Cookbook sre.elasticsearch.rolling-operation (exit_code=97) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: apply kernel updates - bking@cumin2003 - T441838
  • 20:11 bking@cumin2003: START - Cookbook sre.elasticsearch.rolling-operation Operation.REBOOT (1 nodes at a time) for ElasticSearch cluster cloudelastic: apply kernel updates - bking@cumin2003 - T441838
  • 20:10 arlolra@deploy1003: arlolra: Backport for Add "proofread-index" to LINTABLE_CONTENT_MODELS (T441874) (1.48.0-wmf.1, 1.47.0-wmf.24) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 20:09 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1282', diff saved to https://phabricator.wikimedia.org/P96929 and previous config saved to /var/cache/conftool/dbconfig/20261020-200953-ladsgroup.json
  • 20:08 brett@cumin2003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P{cp70[11-16].magru.wmnet} and A:cp
  • 20:08 brett@cumin2003: cookbooks.sre.cdn.roll-reboot finished rebooting cp7016.magru.wmnet
  • 20:06 arlolra@deploy1003: Started scap sync-world: Backport for Add "proofread-index" to LINTABLE_CONTENT_MODELS (T441874) (1.48.0-wmf.1, 1.47.0-wmf.24)
  • 19:59 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1282 (T441633)', diff saved to https://phabricator.wikimedia.org/P96928 and previous config saved to /var/cache/conftool/dbconfig/20261020-195951-ladsgroup.json
  • 19:58 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.reboot (exit_code=0)
  • 19:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1282 (T441633)', diff saved to https://phabricator.wikimedia.org/P96927 and previous config saved to /var/cache/conftool/dbconfig/20261020-195450-ladsgroup.json
  • 19:54 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1282.eqiad.wmnet with reason: Maintenance
  • 19:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1279 (T441633)', diff saved to https://phabricator.wikimedia.org/P96926 and previous config saved to /var/cache/conftool/dbconfig/20261020-195430-ladsgroup.json
  • 19:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1279', diff saved to https://phabricator.wikimedia.org/P96925 and previous config saved to /var/cache/conftool/dbconfig/20261020-194428-ladsgroup.json
  • 19:40 bking@cumin2003: START - Cookbook sre.wdqs.reboot
  • 19:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1279', diff saved to https://phabricator.wikimedia.org/P96924 and previous config saved to /var/cache/conftool/dbconfig/20261020-193426-ladsgroup.json
  • 19:28 brett@cumin2003: cookbooks.sre.cdn.roll-reboot finished rebooting cp7015.magru.wmnet
  • 19:24 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1279 (T441633)', diff saved to https://phabricator.wikimedia.org/P96923 and previous config saved to /var/cache/conftool/dbconfig/20261020-192424-ladsgroup.json
  • 19:23 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.reboot (exit_code=0)
  • 19:19 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1279 (T441633)', diff saved to https://phabricator.wikimedia.org/P96922 and previous config saved to /var/cache/conftool/dbconfig/20261020-191925-ladsgroup.json
  • 19:19 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 6 hosts with reason: Maintenance
  • 19:19 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1279.eqiad.wmnet with reason: Maintenance
  • 19:18 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1241 (T441633)', diff saved to https://phabricator.wikimedia.org/P96921 and previous config saved to /var/cache/conftool/dbconfig/20261020-191844-ladsgroup.json
  • 19:08 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1241', diff saved to https://phabricator.wikimedia.org/P96920 and previous config saved to /var/cache/conftool/dbconfig/20261020-190842-ladsgroup.json
  • 19:06 bking@cumin2003: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: apply kernel updates - bking@cumin2003 - T441838
  • 19:05 bking@cumin2003: START - Cookbook sre.wdqs.reboot
  • 19:05 dancy@deploy1003: Installation of scap version "4.302.0" completed for 158 hosts
  • 19:03 bking@cumin2003: END (FAIL) - Cookbook sre.wdqs.reboot (exit_code=99)
  • 19:03 bking@cumin2003: START - Cookbook sre.wdqs.reboot
  • 19:02 bking@cumin2003: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: apply kernel updates - bking@cumin2003 - T441838
  • 19:01 dancy@deploy1003: Installing scap version "4.302.0" for 158 host(s)
  • 18:58 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1241', diff saved to https://phabricator.wikimedia.org/P96919 and previous config saved to /var/cache/conftool/dbconfig/20261020-185840-ladsgroup.json
  • 18:49 brett@cumin2003: cookbooks.sre.cdn.roll-reboot finished rebooting cp7014.magru.wmnet
  • 18:48 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1241 (T441633)', diff saved to https://phabricator.wikimedia.org/P96918 and previous config saved to /var/cache/conftool/dbconfig/20261020-184838-ladsgroup.json
  • 18:48 brett@cumin2003: END (PASS) - Cookbook sre.cdn.roll-restart-reboot-ncredir (exit_code=0) rolling reboot on A:ncredir
  • 18:41 bking@cumin2003: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: apply kernel updates - bking@cumin2003 - T441838
  • 18:36 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1225.eqiad.wmnet with reason: Maintenance
  • 18:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1201 (T441633)', diff saved to https://phabricator.wikimedia.org/P96916 and previous config saved to /var/cache/conftool/dbconfig/20261020-183542-ladsgroup.json
  • 18:25 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1201', diff saved to https://phabricator.wikimedia.org/P96915 and previous config saved to /var/cache/conftool/dbconfig/20261020-182540-ladsgroup.json
  • 18:24 bking@cumin2003: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.REBOOT (1 nodes at a time) for ElasticSearch cluster relforge: apply kernel updates - bking@cumin2003 - T441838
  • 18:15 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1201', diff saved to https://phabricator.wikimedia.org/P96914 and previous config saved to /var/cache/conftool/dbconfig/20261020-181538-ladsgroup.json
  • 18:14 dancy@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.48.0-wmf.1 refs T438222
  • 18:10 bking@cumin2003: START - Cookbook sre.elasticsearch.rolling-operation Operation.REBOOT (1 nodes at a time) for ElasticSearch cluster relforge: apply kernel updates - bking@cumin2003 - T441838
  • 18:07 brett@cumin2003: cookbooks.sre.cdn.roll-reboot finished rebooting cp7013.magru.wmnet
  • 18:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1201 (T441633)', diff saved to https://phabricator.wikimedia.org/P96912 and previous config saved to /var/cache/conftool/dbconfig/20261020-180536-ladsgroup.json
  • 17:58 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1201 (T441633)', diff saved to https://phabricator.wikimedia.org/P96911 and previous config saved to /var/cache/conftool/dbconfig/20261020-175833-ladsgroup.json
  • 17:58 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1201.eqiad.wmnet with reason: Maintenance
  • 17:58 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1187 (T441633)', diff saved to https://phabricator.wikimedia.org/P96910 and previous config saved to /var/cache/conftool/dbconfig/20261020-175812-ladsgroup.json
  • 17:48 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1187', diff saved to https://phabricator.wikimedia.org/P96909 and previous config saved to /var/cache/conftool/dbconfig/20261020-174811-ladsgroup.json
  • 17:42 dancy@deploy1003: Installation of scap version "4.302.0" completed for 3 hosts
  • 17:41 btullis@cumin1004: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling reboot on A:kafka-test-eqiad
  • 17:41 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.reboot (exit_code=0)
  • 17:40 dancy@deploy1003: Installing scap version "4.302.0" for 3 host(s)
  • 17:39 brett@cumin2003: START - Cookbook sre.cdn.roll-restart-reboot-ncredir rolling reboot on A:ncredir
  • 17:38 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1187', diff saved to https://phabricator.wikimedia.org/P96908 and previous config saved to /var/cache/conftool/dbconfig/20261020-173809-ladsgroup.json
  • 17:30 blake@deploy1003: Finished scap sync-world: bump chart version for T386694 (duration: 04m 54s)
  • 17:28 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1187 (T441633)', diff saved to https://phabricator.wikimedia.org/P96906 and previous config saved to /var/cache/conftool/dbconfig/20261020-172807-ladsgroup.json
  • 17:28 mutante: re-enabled puppet on codesearch1001/2001 - might start to index git repos on gitlab and gerrit
  • 17:26 brett@cumin2003: cookbooks.sre.cdn.roll-reboot finished rebooting cp7012.magru.wmnet
  • 17:26 blake@deploy1003: Started scap sync-world: bump chart version for T386694
  • 17:23 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-worker1142.eqiad.wmnet
  • 17:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1187 (T441633)', diff saved to https://phabricator.wikimedia.org/P96905 and previous config saved to /var/cache/conftool/dbconfig/20261020-172110-ladsgroup.json
  • 17:21 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1187.eqiad.wmnet with reason: Maintenance
  • 17:20 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1168 (T441633)', diff saved to https://phabricator.wikimedia.org/P96904 and previous config saved to /var/cache/conftool/dbconfig/20261020-172050-ladsgroup.json
  • 17:20 bking@cumin2003: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: apply kernel updates - bking@cumin2003 - T441838
  • 17:19 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.reboot (exit_code=0)
  • 17:15 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host an-worker1142.eqiad.wmnet
  • 17:14 btullis@cumin1004: END (FAIL) - Cookbook sre.hadoop.reboot-workers (exit_code=99) for Hadoop analytics cluster
  • 17:14 btullis@cumin1004: START - Cookbook sre.hadoop.reboot-workers for Hadoop analytics cluster
  • 17:13 btullis@cumin1004: END (FAIL) - Cookbook sre.hadoop.reboot-workers (exit_code=99) for Hadoop analytics cluster
  • 17:13 btullis@cumin1004: START - Cookbook sre.hadoop.reboot-workers for Hadoop analytics cluster
  • 17:13 bking@cumin2003: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: apply kernel updates - bking@cumin2003 - T441838
  • 17:13 bking@cumin2003: END (ERROR) - Cookbook sre.elasticsearch.rolling-operation (exit_code=97) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: apply kernel updates - bking@cumin2003 - T441838
  • 17:13 bking@cumin2003: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: apply kernel updates - bking@cumin2003 - T441838
  • 17:10 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P96903 and previous config saved to /var/cache/conftool/dbconfig/20261020-171048-ladsgroup.json
  • 17:06 btullis@cumin1004: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling reboot on A:kafka-test-eqiad
  • 17:05 bking@cumin2003: START - Cookbook sre.wdqs.reboot
  • 17:02 ryankemper: T435477 `[airflow-analytics-test]` upgrade complete; all previously-unpaused DAGs have been unpaused again. ofc, we'll still watch for anything weird that breaks over the next couple days
  • 17:02 bking@cumin2003: END (FAIL) - Cookbook sre.wdqs.reboot (exit_code=99)
  • 17:02 jelto@deploy1003: Finished scap deploy-service miscweb: demo deployment for MVP walk through (duration: 02m 37s)
  • 17:01 jelto@deploy1003: scap deploy-service miscweb: [PRODUCTION/DEFAULT] eqiad Deploying miscweb
  • 17:00 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P96901 and previous config saved to /var/cache/conftool/dbconfig/20261020-170046-ladsgroup.json
  • 17:00 jelto@deploy1003: scap deploy-service miscweb: [PRODUCTION/DEFAULT] codfw Deploying miscweb
  • 16:59 jelto@deploy1003: scap deploy-service miscweb: [STAGING/DEFAULT] staging-eqiad Deploying miscweb
  • 16:59 jelto@deploy1003: Started scap deploy-service miscweb: demo deployment for MVP walk through
  • 16:57 btullis@cumin1004: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling reboot on A:kafka-jumbo-eqiad
  • 16:50 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1168 (T441633)', diff saved to https://phabricator.wikimedia.org/P96900 and previous config saved to /var/cache/conftool/dbconfig/20261020-165043-ladsgroup.json
  • 16:45 brett@cumin2003: cookbooks.sre.cdn.roll-reboot finished rebooting cp7011.magru.wmnet
  • 16:43 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1168 (T441633)', diff saved to https://phabricator.wikimedia.org/P96899 and previous config saved to /var/cache/conftool/dbconfig/20261020-164343-ladsgroup.json
  • 16:43 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1168.eqiad.wmnet with reason: Maintenance
  • 16:43 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1165 (T441633)', diff saved to https://phabricator.wikimedia.org/P96898 and previous config saved to /var/cache/conftool/dbconfig/20261020-164322-ladsgroup.json
  • 16:42 cmooney@cumin1004: END (FAIL) - Cookbook sre.network.tls (exit_code=99) for network device cloudsw3-c8-eqiad
  • 16:42 cmooney@cumin1004: START - Cookbook sre.network.tls for network device cloudsw3-c8-eqiad
  • 16:42 cmooney@cumin1004: END (FAIL) - Cookbook sre.network.tls (exit_code=99) for network device cloudsw3-c8-eqiad
  • 16:42 cmooney@cumin1004: START - Cookbook sre.network.tls for network device cloudsw3-c8-eqiad
  • 16:41 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host druid-internal1006.eqiad.wmnet
  • 16:41 cmooney@cumin1004: END (FAIL) - Cookbook sre.network.tls (exit_code=99) for network device cloudsw3-c8-eqiad
  • 16:41 cmooney@cumin1004: START - Cookbook sre.network.tls for network device cloudsw3-c8-eqiad
  • 16:38 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/lac-lambdas: apply
  • 16:38 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/lac-lambdas: apply
  • 16:37 btullis@cumin1004: END (FAIL) - Cookbook sre.hadoop.reboot-workers (exit_code=99) for Hadoop analytics cluster
  • 16:37 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 16:37 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add dns entries new cloudsw - cmooney@cumin1004"
  • 16:37 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add dns entries new cloudsw - cmooney@cumin1004"
  • 16:36 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host druid-internal1006.eqiad.wmnet
  • 16:33 brett@cumin2003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P{cp70[11-16].magru.wmnet} and A:cp
  • 16:33 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P96897 and previous config saved to /var/cache/conftool/dbconfig/20261020-163321-ladsgroup.json
  • 16:32 cmooney@cumin1004: START - Cookbook sre.dns.netbox
  • 16:31 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host druid-internal1005.eqiad.wmnet
  • 16:29 bking@cumin2003: START - Cookbook sre.wdqs.reboot
  • 16:28 bking@cumin2003: END (FAIL) - Cookbook sre.wdqs.reboot (exit_code=99)
  • 16:28 bking@cumin2003: START - Cookbook sre.wdqs.reboot
  • 16:27 bking@cumin2003: START - Cookbook sre.wdqs.reboot
  • 16:25 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host druid-internal1005.eqiad.wmnet
  • 16:23 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P96895 and previous config saved to /var/cache/conftool/dbconfig/20261020-162319-ladsgroup.json
  • 16:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1165 (T441633)', diff saved to https://phabricator.wikimedia.org/P96893 and previous config saved to /var/cache/conftool/dbconfig/20261020-161317-ladsgroup.json
  • 16:12 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.reboot (exit_code=0)
  • 16:11 ryankemper: T435477 `[airflow-analytics-test]` applying https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1357219, so this marks the official cutover
  • 16:10 ryankemper@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply
  • 16:09 ryankemper@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply
  • 16:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1165 (T441633)', diff saved to https://phabricator.wikimedia.org/P96892 and previous config saved to /var/cache/conftool/dbconfig/20261020-160606-ladsgroup.json
  • 16:06 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1165.eqiad.wmnet with reason: Maintenance
  • 15:59 trueg@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply
  • 15:59 herron@cumin1004: START - Cookbook sre.opensearch.roll-restart-reboot rolling reboot on A:logstash-eqiad
  • 15:55 trueg@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply
  • 15:55 trueg@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply
  • 15:54 ryankemper: T435477 `[airflow-analytics-test]` all associated DAGs paused
  • 15:53 marostegui@dns1004: END - running authdns-update
  • 15:52 trueg@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply
  • 15:51 marostegui@dns1004: START - running authdns-update
  • 15:48 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply
  • 15:48 ryankemper: T435477 `[airflow-analytics-test]` gearing up to start upgrade to Airflow 3.3, pausing all associated DAGs shortly
  • 15:48 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply
  • 15:48 mvernon@cumin1004: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe
  • 15:47 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply
  • 15:46 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply
  • 15:44 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
  • 15:43 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
  • 15:42 trueg@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply
  • 15:41 dancy@deploy1003: Finished scap sync-world: testwikis to 1.48.0-wmf.1 refs T438222 (duration: 37m 07s)
  • 15:36 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe
  • 15:32 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe
  • 15:32 trueg@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply
  • 15:31 mvernon@cumin1004: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe
  • 15:31 trueg@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply
  • 15:31 trueg@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply
  • 15:27 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host druid-internal1004.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 15:23 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
  • 15:23 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
  • 15:21 btullis@cumin1004: START - Cookbook sre.hosts.provision for host druid-internal1004.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 15:13 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) pooling P{lvs7002.*} and A:liberica
  • 15:13 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin pooling P{lvs7002.*} and A:liberica
  • 15:05 elukey@cumin1004: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling restart_daemons on A:kafka-main-eqiad
  • 15:05 bking@cumin2003: START - Cookbook sre.wdqs.reboot
  • 15:04 dancy@deploy1003: Started scap sync-world: testwikis to 1.48.0-wmf.1 refs T438222
  • 15:00 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:ml-staging-worker
  • 15:00 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-staging2003.codfw.wmnet
  • 15:00 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host ml-staging2003.codfw.wmnet
  • 14:58 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.reboot (exit_code=0)
  • 14:57 cscott@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
  • 14:57 cscott@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
  • 14:57 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) depooling P{lvs7002.*} and A:liberica
  • 14:57 effie: depooled eqiad for mw-web T440896
  • 14:57 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin depooling P{lvs7002.*} and A:liberica
  • 14:55 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-staging2003.codfw.wmnet
  • 14:52 jiji@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) depool mw-web-ro in eqiad: T440896
  • 14:52 dkertesz@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-text_magru and not P{cp7001.*} and A:cp - upgrade haproxy to 3.2.26-1 (T441435)
  • 14:48 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.reboot (exit_code=0)
  • 14:47 jiji@cumin1004: START - Cookbook sre.discovery.service-route depool mw-web-ro in eqiad: T440896
  • 14:45 elukey@cumin1004: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling restart_daemons on A:kafka-main-eqiad
  • 14:45 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host ml-staging2003.codfw.wmnet
  • 14:45 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-staging2002.codfw.wmnet
  • 14:45 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host ml-staging2002.codfw.wmnet
  • 14:44 elukey@cumin1004: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling restart_daemons on A:kafka-main-codfw
  • 14:43 bking@cumin2003: START - Cookbook sre.wdqs.reboot
  • 14:41 vgutierrez: upload ipip-multiqueue-optimizer 0.3+deb13u1 to trixie-wikimedia (apt.wm.o) - T401832
  • 14:40 vgutierrez@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host lvs7003.magru.wmnet with OS trixie
  • 14:40 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-staging2002.codfw.wmnet
  • 14:39 bking@cumin2003: START - Cookbook sre.wdqs.reboot
  • 14:38 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.reboot (exit_code=0)
  • 14:37 cscott@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
  • 14:37 cscott@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
  • 14:37 trueg@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply
  • 14:37 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.reboot (exit_code=0)
  • 14:34 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-web: apply
  • 14:34 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-web: apply
  • 14:32 elukey@cumin1004: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling restart_daemons on A:kafka-logging-eqiad
  • 14:30 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host ml-staging2002.codfw.wmnet
  • 14:30 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-staging2001.codfw.wmnet
  • 14:30 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host ml-staging2001.codfw.wmnet
  • 14:30 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply
  • 14:29 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-ext: apply
  • 14:29 bking@cumin2003: START - Cookbook sre.wdqs.reboot
  • 14:29 bking@cumin2003: START - Cookbook sre.wdqs.reboot
  • 14:29 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.sync-bgp-netbox (exit_code=0)
  • 14:29 ayounsi@cumin1004: START - Cookbook sre.network.sync-bgp-netbox
  • 14:29 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.reboot (exit_code=0)
  • 14:26 trueg@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply
  • 14:25 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.reboot (exit_code=0)
  • 14:25 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-staging2001.codfw.wmnet
  • 14:24 elukey@cumin1004: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling restart_daemons on A:kafka-main-codfw
  • 14:22 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply
  • 14:20 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'llmisvc' .
  • 14:20 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
  • 14:20 bking@cumin2003: START - Cookbook sre.wdqs.reboot
  • 14:19 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply
  • 14:19 bking@cumin2003: START - Cookbook sre.wdqs.reboot
  • 14:19 bking@cumin2003: END (FAIL) - Cookbook sre.wdqs.reboot (exit_code=99)
  • 14:18 bking@cumin2003: START - Cookbook sre.wdqs.reboot
  • 14:18 bking@cumin2003: END (FAIL) - Cookbook sre.wdqs.reboot (exit_code=99)
  • 14:18 bking@cumin2003: START - Cookbook sre.wdqs.reboot
  • 14:17 hashar@deploy1003: Finished deploy [integration/docroot@be38d5a]: opensource: docker-pkg has moved to GitLab - T441088 (duration: 00m 10s)
  • 14:17 hashar@deploy1003: Started deploy [integration/docroot@be38d5a]: opensource: docker-pkg has moved to GitLab - T441088
  • 14:15 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host ml-staging2001.codfw.wmnet
  • 14:15 klausman@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:ml-staging-worker
  • 14:13 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.reboot (exit_code=0)
  • 14:13 klausman@cumin1004: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-serve-ctrl2002.codfw.wmnet
  • 14:12 elukey@cumin1004: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling restart_daemons on A:kafka-logging-eqiad
  • 14:11 elukey@cumin1004: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling restart_daemons on A:kafka-logging-codfw
  • 14:10 hashar@deploy1003: Finished deploy [integration/docroot@5cf09c8]: opensource: docker-pkg has moved to GitLab - T441088 (duration: 00m 10s)
  • 14:10 hashar@deploy1003: Started deploy [integration/docroot@5cf09c8]: opensource: docker-pkg has moved to GitLab - T441088
  • 14:10 sukhe@cumin1004: END (PASS) - Cookbook sre.dns.roll-reboot (exit_code=0) rolling reboot on P{dns7002* or dns6001* or dns4003*} and (A:dnsbox)
  • 14:10 sukhe@cumin1004: cookbooks.sre.dns.roll-reboot finished rebooting dns7002.wikimedia.org
  • 14:10 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply
  • 14:08 vgutierrez@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs7003.magru.wmnet with reason: host reimage
  • 14:08 klausman@cumin1004: START - Cookbook sre.ganeti.reboot-vm for VM ml-serve-ctrl2002.codfw.wmnet
  • 14:08 klausman@cumin1004: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-serve-ctrl2001.codfw.wmnet
  • 14:07 bking@cumin2003: START - Cookbook sre.wdqs.reboot
  • 14:06 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply
  • 14:04 a-pizzata@deploy1003: Finished deploy [analytics/refinery@68687dc]: Regular analytics weekly train [analytics/refinery@68687dc4] (duration: 05m 39s)
  • 14:03 vgutierrez@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on lvs7003.magru.wmnet with reason: host reimage
  • 14:03 klausman@cumin1004: START - Cookbook sre.ganeti.reboot-vm for VM ml-serve-ctrl2001.codfw.wmnet
  • 14:02 klausman@cumin1004: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-serve-ctrl1002.eqiad.wmnet
  • 14:00 sukhe@cumin1004: cookbooks.sre.dns.roll-reboot begin reboot of dns7002.wikimedia.org
  • 13:59 a-pizzata@deploy1003: Started deploy [analytics/refinery@68687dc]: Regular analytics weekly train [analytics/refinery@68687dc4]
  • 13:58 a-pizzata@deploy1003: Finished deploy [analytics/refinery@68687dc] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@68687dc4] (duration: 00m 46s)
  • 13:58 klausman@cumin1004: START - Cookbook sre.ganeti.reboot-vm for VM ml-serve-ctrl1002.eqiad.wmnet
  • 13:58 klausman@cumin1004: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-serve-ctrl1001.eqiad.wmnet
  • 13:58 a-pizzata@deploy1003: Started deploy [analytics/refinery@68687dc] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@68687dc4]
  • 13:55 klausman@cumin1004: START - Cookbook sre.ganeti.reboot-vm for VM ml-serve-ctrl1001.eqiad.wmnet
  • 13:54 klausman@cumin1004: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-etcd1003.eqiad.wmnet
  • 13:54 elukey@cumin1004: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling restart_daemons on A:kafka-logging-codfw
  • 13:52 klausman@cumin1004: START - Cookbook sre.ganeti.reboot-vm for VM ml-etcd1003.eqiad.wmnet
  • 13:51 dkertesz@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_magru and not P{cp7001.*} and A:cp - upgrade haproxy to 3.2.26-1 (T441435)
  • 13:51 klausman@cumin1004: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-etcd1002.eqiad.wmnet
  • 13:49 klausman@cumin1004: START - Cookbook sre.ganeti.reboot-vm for VM ml-etcd1002.eqiad.wmnet
  • 13:49 klausman@cumin1004: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-etcd1001.eqiad.wmnet
  • 13:47 klausman@cumin1004: START - Cookbook sre.ganeti.reboot-vm for VM ml-etcd1001.eqiad.wmnet
  • 13:46 elukey@cumin1004: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling restart_daemons on A:kafka-test-eqiad
  • 13:46 klausman@cumin1004: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-etcd2003.codfw.wmnet
  • 13:45 sukhe@cumin1004: cookbooks.sre.dns.roll-reboot finished rebooting dns6001.wikimedia.org
  • 13:43 klausman@cumin1004: START - Cookbook sre.ganeti.reboot-vm for VM ml-etcd2003.codfw.wmnet
  • 13:43 klausman@cumin1004: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-etcd2002.codfw.wmnet
  • 13:41 klausman@cumin1004: START - Cookbook sre.ganeti.reboot-vm for VM ml-etcd2002.codfw.wmnet
  • 13:40 vgutierrez@cumin1004: START - Cookbook sre.hosts.reimage for host lvs7003.magru.wmnet with OS trixie
  • 13:39 klausman@cumin1004: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-etcd2001.codfw.wmnet
  • 13:39 ayounsi@cumin1004: END (FAIL) - Cookbook sre.network.peering (exit_code=99) with action 'configure' for AS: 44034
  • 13:38 ayounsi@cumin1004: START - Cookbook sre.network.peering with action 'configure' for AS: 44034
  • 13:37 vgutierrez: upload liberica 0.28 to trixie-wikimedia (apt.wm.o) - T401832
  • 13:35 klausman@cumin1004: START - Cookbook sre.ganeti.reboot-vm for VM ml-etcd2001.codfw.wmnet
  • 13:35 klausman@cumin1004: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-staging-ctrl2002.codfw.wmnet
  • 13:31 klausman@cumin1004: START - Cookbook sre.ganeti.reboot-vm for VM ml-staging-ctrl2002.codfw.wmnet
  • 13:31 klausman@cumin1004: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-staging-ctrl2001.codfw.wmnet
  • 13:30 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host druid-internal1003.eqiad.wmnet
  • 13:29 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 18 hosts with reason: Restart
  • 13:29 sukhe@cumin1004: cookbooks.sre.dns.roll-reboot begin reboot of dns6001.wikimedia.org
  • 13:27 klausman@cumin1004: START - Cookbook sre.ganeti.reboot-vm for VM ml-staging-ctrl2001.codfw.wmnet
  • 13:27 klausman@cumin1004: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-staging-etcd2003.codfw.wmnet
  • 13:27 elukey@cumin1004: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling restart_daemons on A:kafka-test-eqiad
  • 13:25 klausman@cumin1004: START - Cookbook sre.ganeti.reboot-vm for VM ml-staging-etcd2003.codfw.wmnet
  • 13:25 klausman@cumin1004: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-staging-etcd2002.codfw.wmnet
  • 13:25 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host druid-internal1003.eqiad.wmnet
  • 13:25 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host druid-internal1002.eqiad.wmnet
  • 13:22 klausman@cumin1004: START - Cookbook sre.ganeti.reboot-vm for VM ml-staging-etcd2002.codfw.wmnet
  • 13:22 klausman@cumin1004: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-staging-etcd2001.codfw.wmnet
  • 13:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2224 (T441633)', diff saved to https://phabricator.wikimedia.org/P96889 and previous config saved to /var/cache/conftool/dbconfig/20261020-132118-ladsgroup.json
  • 13:20 klausman@cumin1004: START - Cookbook sre.ganeti.reboot-vm for VM ml-staging-etcd2001.codfw.wmnet
  • 13:19 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host druid-internal1002.eqiad.wmnet
  • 13:16 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudrabbit1003.eqiad.wmnet
  • 13:14 sukhe@cumin1004: cookbooks.sre.dns.roll-reboot finished rebooting dns4003.wikimedia.org
  • 13:14 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host druid-internal1001.eqiad.wmnet
  • 13:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2224', diff saved to https://phabricator.wikimedia.org/P96888 and previous config saved to /var/cache/conftool/dbconfig/20261020-131116-ladsgroup.json
  • 13:10 filippo@cumin1004: START - Cookbook sre.hosts.reboot-single for host cloudrabbit1003.eqiad.wmnet
  • 13:09 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudrabbit1002.eqiad.wmnet
  • 13:09 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host druid-internal1001.eqiad.wmnet
  • 13:03 filippo@cumin1004: START - Cookbook sre.hosts.reboot-single for host cloudrabbit1002.eqiad.wmnet
  • 13:01 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudrabbit1001.eqiad.wmnet
  • 13:01 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2224', diff saved to https://phabricator.wikimedia.org/P96887 and previous config saved to /var/cache/conftool/dbconfig/20261020-130114-ladsgroup.json
  • 13:00 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-mcrouter: apply
  • 13:00 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-mcrouter: apply
  • 12:58 sukhe@cumin1004: cookbooks.sre.dns.roll-reboot begin reboot of dns4003.wikimedia.org
  • 12:58 sukhe@cumin1004: START - Cookbook sre.dns.roll-reboot rolling reboot on P{dns7002* or dns6001* or dns4003*} and (A:dnsbox)
  • 12:55 filippo@cumin1004: START - Cookbook sre.hosts.reboot-single for host cloudrabbit1001.eqiad.wmnet
  • 12:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2224 (T441633)', diff saved to https://phabricator.wikimedia.org/P96886 and previous config saved to /var/cache/conftool/dbconfig/20261020-125112-ladsgroup.json
  • 12:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2224 (T441633)', diff saved to https://phabricator.wikimedia.org/P96885 and previous config saved to /var/cache/conftool/dbconfig/20261020-124421-ladsgroup.json
  • 12:44 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2224.codfw.wmnet with reason: Maintenance
  • 12:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2217 (T441633)', diff saved to https://phabricator.wikimedia.org/P96884 and previous config saved to /var/cache/conftool/dbconfig/20261020-124359-ladsgroup.json
  • 12:37 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dbstore1010.eqiad.wmnet
  • 12:33 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2217', diff saved to https://phabricator.wikimedia.org/P96883 and previous config saved to /var/cache/conftool/dbconfig/20261020-123357-ladsgroup.json
  • 12:32 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dbstore1010.eqiad.wmnet
  • 12:29 btullis@cumin1004: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling reboot on A:kafka-jumbo-eqiad
  • 12:28 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ceph-admin1001.eqiad.wmnet
  • 12:28 cwilliams@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host backup1014.eqiad.wmnet
  • 12:24 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host ceph-admin1001.eqiad.wmnet
  • 12:24 cwilliams@cumin1004: START - Cookbook sre.hosts.reboot-single for host backup1014.eqiad.wmnet
  • 12:23 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2217', diff saved to https://phabricator.wikimedia.org/P96882 and previous config saved to /var/cache/conftool/dbconfig/20261020-122356-ladsgroup.json
  • 12:21 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudvirtlocal1003.eqiad.wmnet
  • 12:20 cwilliams@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host backup2014.codfw.wmnet
  • 12:16 cwilliams@cumin1004: START - Cookbook sre.hosts.reboot-single for host backup2014.codfw.wmnet
  • 12:16 cwilliams@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host backup2012.codfw.wmnet
  • 12:14 filippo@cumin1004: START - Cookbook sre.hosts.reboot-single for host cloudvirtlocal1003.eqiad.wmnet
  • 12:14 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on dbproxy1023.eqiad.wmnet with reason: Reboot
  • 12:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2217 (T441633)', diff saved to https://phabricator.wikimedia.org/P96881 and previous config saved to /var/cache/conftool/dbconfig/20261020-121354-ladsgroup.json
  • 12:13 cwilliams@cumin1004: START - Cookbook sre.hosts.reboot-single for host backup2012.codfw.wmnet
  • 12:13 cwilliams@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host backup2009.codfw.wmnet
  • 12:11 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudvirtlocal1002.eqiad.wmnet
  • 12:11 cwilliams@cumin1004: START - Cookbook sre.hosts.reboot-single for host backup2009.codfw.wmnet
  • 12:10 cwilliams@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host backup2008.codfw.wmnet
  • 12:10 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on dbproxy1024.eqiad.wmnet with reason: Reboot
  • 12:07 cwilliams@cumin1004: START - Cookbook sre.hosts.reboot-single for host backup2008.codfw.wmnet
  • 12:07 cwilliams@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host backup1012.eqiad.wmnet
  • 12:07 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2217 (T441633)', diff saved to https://phabricator.wikimedia.org/P96880 and previous config saved to /var/cache/conftool/dbconfig/20261020-120701-ladsgroup.json
  • 12:07 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2217.codfw.wmnet with reason: Maintenance
  • 12:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2214 (T441633)', diff saved to https://phabricator.wikimedia.org/P96879 and previous config saved to /var/cache/conftool/dbconfig/20261020-120640-ladsgroup.json
  • 12:04 filippo@cumin1004: START - Cookbook sre.hosts.reboot-single for host cloudvirtlocal1002.eqiad.wmnet
  • 12:03 jiji@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) depool mw-api-ext-ro in eqiad: T440896
  • 12:02 cwilliams@cumin1004: START - Cookbook sre.hosts.reboot-single for host backup1012.eqiad.wmnet
  • 12:02 cwilliams@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host backup1009.eqiad.wmnet
  • 12:00 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
  • 12:00 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
  • 11:59 cwilliams@cumin1004: START - Cookbook sre.hosts.reboot-single for host backup1009.eqiad.wmnet
  • 11:59 cwilliams@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host backup1008.eqiad.wmnet
  • 11:58 jiji@cumin1004: START - Cookbook sre.discovery.service-route depool mw-api-ext-ro in eqiad: T440896
  • 11:56 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2214', diff saved to https://phabricator.wikimedia.org/P96878 and previous config saved to /var/cache/conftool/dbconfig/20261020-115638-ladsgroup.json
  • 11:56 cwilliams@cumin1004: START - Cookbook sre.hosts.reboot-single for host backup1008.eqiad.wmnet
  • 11:54 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudvirtlocal1001.eqiad.wmnet
  • 11:49 btullis@cumin1004: END (PASS) - Cookbook sre.apifeatureusage.roll-restart-reboot-logstash (exit_code=0) rolling reboot on A:apifeatureusage
  • 11:47 filippo@cumin1004: START - Cookbook sre.hosts.reboot-single for host cloudvirtlocal1001.eqiad.wmnet
  • 11:46 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2214', diff saved to https://phabricator.wikimedia.org/P96877 and previous config saved to /var/cache/conftool/dbconfig/20261020-114636-ladsgroup.json
  • 11:45 slyngshede@cumin1004: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on A:cp-text_magru and not P{cp7002*} and not P{cp7010*} and A:cp
  • 11:45 slyngshede@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp7009.magru.wmnet
  • 11:45 cwilliams@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on 8 hosts with reason: Maintenance
  • 11:44 cwilliams@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on backup1014.eqiad.wmnet with reason: Maintenance
  • 11:43 btullis@cumin1004: END (PASS) - Cookbook sre.ceph.roll-restart-reboot-server (exit_code=0) rolling reboot on A:cephosd-eqiad
  • 11:38 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'llmisvc' .
  • 11:37 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
  • 11:37 btullis@cumin1004: START - Cookbook sre.apifeatureusage.roll-restart-reboot-logstash rolling reboot on A:apifeatureusage
  • 11:36 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2214 (T441633)', diff saved to https://phabricator.wikimedia.org/P96876 and previous config saved to /var/cache/conftool/dbconfig/20261020-113634-ladsgroup.json
  • 11:35 kevinbazira@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
  • 11:35 kevinbazira@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
  • 11:35 kamila@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
  • 11:29 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2214 (T441633)', diff saved to https://phabricator.wikimedia.org/P96875 and previous config saved to /var/cache/conftool/dbconfig/20261020-112940-ladsgroup.json
  • 11:29 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2214.codfw.wmnet with reason: Maintenance
  • 11:28 kamila@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
  • 11:28 kamila@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
  • 11:28 kamila@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
  • 11:25 vgutierrez@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host acmechief2002.codfw.wmnet
  • 11:25 vgutierrez@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host acmechief1002.eqiad.wmnet
  • 11:23 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2197.codfw.wmnet with reason: Maintenance
  • 11:23 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2193 (T441633)', diff saved to https://phabricator.wikimedia.org/P96874 and previous config saved to /var/cache/conftool/dbconfig/20261020-112333-ladsgroup.json
  • 11:21 vgutierrez@cumin1004: START - Cookbook sre.hosts.reboot-single for host acmechief2002.codfw.wmnet
  • 11:21 vgutierrez@cumin1004: START - Cookbook sre.hosts.reboot-single for host acmechief1002.eqiad.wmnet
  • 11:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2193', diff saved to https://phabricator.wikimedia.org/P96873 and previous config saved to /var/cache/conftool/dbconfig/20261020-111332-ladsgroup.json
  • 11:10 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1032.eqiad.wmnet
  • 11:10 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2030.codfw.wmnet
  • 11:09 jiji@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) depool mw-api-int-ro in eqiad: T440896
  • 11:09 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1032.eqiad.wmnet
  • 11:09 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2030.codfw.wmnet
  • 11:08 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P{lvs5005.*,lvs4009.*} and A:liberica
  • 11:05 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti-test2003.codfw.wmnet
  • 11:04 jiji@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) mw-api-int-ro.discovery.wmnet on all recursors
  • 11:04 jiji@cumin1004: START - Cookbook sre.dns.wipe-cache mw-api-int-ro.discovery.wmnet on all recursors
  • 11:04 jiji@cumin1004: START - Cookbook sre.discovery.service-route depool mw-api-int-ro in eqiad: T440896
  • 11:04 slyngshede@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp7008.magru.wmnet
  • 11:03 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti1032.eqiad.wmnet
  • 11:03 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2193', diff saved to https://phabricator.wikimedia.org/P96872 and previous config saved to /var/cache/conftool/dbconfig/20261020-110330-ladsgroup.json
  • 11:03 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti2030.codfw.wmnet
  • 11:01 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin rebooting P{lvs5005.*,lvs4009.*} and A:liberica
  • 11:00 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1032.eqiad.wmnet
  • 10:59 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2030.codfw.wmnet
  • 10:59 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti-test2003.codfw.wmnet
  • 10:59 kamila@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
  • 10:58 kamila@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
  • 10:58 kamila@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
  • 10:57 kamila@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
  • 10:53 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2193 (T441633)', diff saved to https://phabricator.wikimedia.org/P96871 and previous config saved to /var/cache/conftool/dbconfig/20261020-105328-ladsgroup.json
  • 10:50 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host archiva1002.wikimedia.org
  • 10:50 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
  • 10:48 arnaudb@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'.
  • 10:47 arnaudb@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'.
  • 10:47 vgutierrez@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host acmechief-test2001.codfw.wmnet
  • 10:46 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host archiva1002.wikimedia.org
  • 10:46 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2193 (T441633)', diff saved to https://phabricator.wikimedia.org/P96870 and previous config saved to /var/cache/conftool/dbconfig/20261020-104634-ladsgroup.json
  • 10:46 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2193.codfw.wmnet with reason: Maintenance
  • 10:46 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2180 (T441633)', diff saved to https://phabricator.wikimedia.org/P96869 and previous config saved to /var/cache/conftool/dbconfig/20261020-104623-ladsgroup.json
  • 10:46 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host krb1005.eqiad.wmnet
  • 10:44 root@cumin2003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-eqiad
  • 10:44 root@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker1009.eqiad.wmnet
  • 10:44 root@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1009.eqiad.wmnet
  • 10:43 vgutierrez@cumin1004: START - Cookbook sre.hosts.reboot-single for host acmechief-test2001.codfw.wmnet
  • 10:43 vgutierrez@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host acmechief-test1001.eqiad.wmnet
  • 10:42 arnaudb@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
  • 10:42 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host krb1005.eqiad.wmnet
  • 10:42 arnaudb@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
  • 10:40 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host matomo1003.eqiad.wmnet
  • 10:39 vgutierrez@cumin1004: START - Cookbook sre.hosts.reboot-single for host acmechief-test1001.eqiad.wmnet
  • 10:39 root@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1009.eqiad.wmnet
  • 10:38 root@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1009.eqiad.wmnet
  • 10:38 root@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker1008.eqiad.wmnet
  • 10:38 root@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1008.eqiad.wmnet
  • 10:36 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host matomo1003.eqiad.wmnet
  • 10:36 jiji@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) check mw-api-ext-ro: maintenance
  • 10:36 jiji@cumin1004: START - Cookbook sre.discovery.service-route check mw-api-ext-ro: maintenance
  • 10:36 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2180', diff saved to https://phabricator.wikimedia.org/P96868 and previous config saved to /var/cache/conftool/dbconfig/20261020-103621-ladsgroup.json
  • 10:36 jiji@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) check mw-api-int-ro: maintenance
  • 10:36 jiji@cumin1004: START - Cookbook sre.discovery.service-route check mw-api-int-ro: maintenance
  • 10:35 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ceph-admin2001.codfw.wmnet
  • 10:33 root@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1008.eqiad.wmnet
  • 10:32 root@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1008.eqiad.wmnet
  • 10:32 root@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker1007.eqiad.wmnet
  • 10:32 root@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1007.eqiad.wmnet
  • 10:31 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host ceph-admin2001.codfw.wmnet
  • 10:28 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) pooling P{lvs7001.*} and A:liberica
  • 10:28 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin pooling P{lvs7001.*} and A:liberica
  • 10:27 vgutierrez@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host lvs7001.magru.wmnet
  • 10:27 root@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1007.eqiad.wmnet
  • 10:26 root@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1007.eqiad.wmnet
  • 10:26 root@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker1006.eqiad.wmnet
  • 10:26 root@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1006.eqiad.wmnet
  • 10:26 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2180', diff saved to https://phabricator.wikimedia.org/P96867 and previous config saved to /var/cache/conftool/dbconfig/20261020-102619-ladsgroup.json
  • 10:26 btullis@cumin1004: START - Cookbook sre.ceph.roll-restart-reboot-server rolling reboot on A:cephosd-eqiad
  • 10:24 marostegui@dns1004: END - running authdns-update
  • 10:24 vgutierrez@cumin1004: START - Cookbook sre.hosts.reboot-single for host lvs7001.magru.wmnet
  • 10:23 vgutierrez@cumin1004: END (FAIL) - Cookbook sre.loadbalancer.admin (exit_code=1) rebooting P{lvs7001.*} and A:liberica
  • 10:23 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin rebooting P{lvs7001.*} and A:liberica
  • 10:22 slyngshede@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp7007.magru.wmnet
  • 10:22 marostegui@dns1004: START - running authdns-update
  • 10:21 root@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1006.eqiad.wmnet
  • 10:21 root@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1006.eqiad.wmnet
  • 10:21 root@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker1005.eqiad.wmnet
  • 10:20 root@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1005.eqiad.wmnet
  • 10:17 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti-test2002.codfw.wmnet
  • 10:16 cwilliams@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host backupmon1001.eqiad.wmnet
  • 10:15 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1031.eqiad.wmnet
  • 10:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1031.eqiad.wmnet
  • 10:14 root@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1005.eqiad.wmnet
  • 10:13 root@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker1004.eqiad.wmnet
  • 10:13 root@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1004.eqiad.wmnet
  • 10:12 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2029.codfw.wmnet
  • 10:11 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2029.codfw.wmnet
  • 10:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2180 (T441633)', diff saved to https://phabricator.wikimedia.org/P96865 and previous config saved to /var/cache/conftool/dbconfig/20261020-100926-ladsgroup.json
  • 10:10 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2180.codfw.wmnet with reason: Maintenance
  • 10:09 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2158 (T441633)', diff saved to https://phabricator.wikimedia.org/P96864 and previous config saved to /var/cache/conftool/dbconfig/20261020-100904-ladsgroup.json
  • 10:09 cwilliams@cumin1004: START - Cookbook sre.hosts.reboot-single for host backupmon1001.eqiad.wmnet
  • 10:07 root@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1004.eqiad.wmnet
  • 10:07 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti-test2002.codfw.wmnet
  • 10:06 root@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1004.eqiad.wmnet
  • 10:06 root@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker1003.eqiad.wmnet
  • 10:06 root@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1003.eqiad.wmnet
  • 10:06 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti1031.eqiad.wmnet
  • 10:05 cwilliams@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host backup2011.codfw.wmnet
  • 10:04 moritzm: powercycle ganeti2029
  • 10:03 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1031.eqiad.wmnet
  • 10:03 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1030.eqiad.wmnet
  • 10:02 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1030.eqiad.wmnet
  • 10:02 root@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1003.eqiad.wmnet
  • 10:02 cwilliams@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host backup1011.eqiad.wmnet
  • 10:02 root@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1003.eqiad.wmnet
  • 10:02 root@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker1002.eqiad.wmnet
  • 10:01 root@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1002.eqiad.wmnet
  • 09:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host debmonitor1003.eqiad.wmnet
  • 09:59 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2158', diff saved to https://phabricator.wikimedia.org/P96863 and previous config saved to /var/cache/conftool/dbconfig/20261020-095902-ladsgroup.json
  • 09:58 cwilliams@cumin1004: START - Cookbook sre.hosts.reboot-single for host backup2011.codfw.wmnet
  • 09:58 cwilliams@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host backup2010.codfw.wmnet
  • 09:58 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti1030.eqiad.wmnet
  • 09:57 root@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1002.eqiad.wmnet
  • 09:57 root@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1002.eqiad.wmnet
  • 09:57 root@cumin2003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-eqiad
  • 09:55 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1030.eqiad.wmnet
  • 09:55 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti2029.codfw.wmnet
  • 09:55 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host debmonitor1003.eqiad.wmnet
  • 09:55 cwilliams@cumin1004: START - Cookbook sre.hosts.reboot-single for host backup1011.eqiad.wmnet
  • 09:55 cwilliams@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host backup1010.eqiad.wmnet
  • 09:55 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1029.eqiad.wmnet
  • 09:54 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1029.eqiad.wmnet
  • 09:53 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P{lvs7002.*} and A:liberica
  • 09:52 ayounsi@cumin1004: END (FAIL) - Cookbook sre.network.peering (exit_code=99) with action 'configure' for AS: 3292
  • 09:52 root@cumin2003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-codfw
  • 09:52 root@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker2009.codfw.wmnet
  • 09:51 root@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker2009.codfw.wmnet
  • 09:51 ayounsi@cumin1004: START - Cookbook sre.network.peering with action 'configure' for AS: 3292
  • 09:51 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host debmonitor2003.codfw.wmnet
  • 09:50 cwilliams@cumin1004: START - Cookbook sre.hosts.reboot-single for host backup2010.codfw.wmnet
  • 09:50 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti1029.eqiad.wmnet
  • 09:50 jelto@deploy1003: Finished scap deploy-service miscweb: demo deployment for MVP walk through (duration: 03m 59s)
  • 09:49 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin rebooting P{lvs7002.*} and A:liberica
  • 09:49 cwilliams@cumin1004: START - Cookbook sre.hosts.reboot-single for host backup1010.eqiad.wmnet
  • 09:49 jelto@deploy1003: scap deploy-service miscweb: [PRODUCTION/DEFAULT] eqiad Deploying miscweb
  • 09:49 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2158', diff saved to https://phabricator.wikimedia.org/P96862 and previous config saved to /var/cache/conftool/dbconfig/20261020-094901-ladsgroup.json
  • 09:47 jelto@deploy1003: scap deploy-service miscweb: [PRODUCTION/DEFAULT] codfw Deploying miscweb
  • 09:47 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2029.codfw.wmnet
  • 09:47 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host debmonitor2003.codfw.wmnet
  • 09:46 root@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker2009.codfw.wmnet
  • 09:46 jelto@deploy1003: scap deploy-service miscweb: [STAGING/DEFAULT] staging-eqiad Deploying miscweb
  • 09:46 jelto@deploy1003: Started scap deploy-service miscweb: demo deployment for MVP walk through
  • 09:45 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host debmonitor-dev2001.codfw.wmnet
  • 09:44 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2028.codfw.wmnet
  • 09:44 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2028.codfw.wmnet
  • 09:43 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1029.eqiad.wmnet
  • 09:41 root@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker2009.codfw.wmnet
  • 09:41 root@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker2008.codfw.wmnet
  • 09:41 root@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker2008.codfw.wmnet
  • 09:41 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host debmonitor-dev2001.codfw.wmnet
  • 09:40 slyngshede@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp7006.magru.wmnet
  • 09:39 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2158 (T441633)', diff saved to https://phabricator.wikimedia.org/P96860 and previous config saved to /var/cache/conftool/dbconfig/20261020-093859-ladsgroup.json
  • 09:38 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti2028.codfw.wmnet
  • 09:36 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2028.codfw.wmnet
  • 09:35 root@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker2008.codfw.wmnet
  • 09:35 root@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker2008.codfw.wmnet
  • 09:35 root@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker2007.codfw.wmnet
  • 09:35 root@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker2007.codfw.wmnet
  • 09:32 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2158 (T441633)', diff saved to https://phabricator.wikimedia.org/P96859 and previous config saved to /var/cache/conftool/dbconfig/20261020-093218-ladsgroup.json
  • 09:32 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2158.codfw.wmnet with reason: Maintenance
  • 09:30 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host irc2003.wikimedia.org
  • 09:30 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2027.codfw.wmnet
  • 09:30 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2027.codfw.wmnet
  • 09:30 btullis@cumin1004: END (FAIL) - Cookbook sre.ceph.roll-restart-reboot-server (exit_code=1) rolling reboot on A:cephosd-eqiad
  • 09:30 hashar: restarting MediaWiki branch cutting job # T438222 T441774
  • 09:29 root@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker2007.codfw.wmnet
  • 09:29 root@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker2007.codfw.wmnet
  • 09:29 root@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker2006.codfw.wmnet
  • 09:29 root@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker2006.codfw.wmnet
  • 09:26 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host irc2003.wikimedia.org
  • 09:24 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti2027.codfw.wmnet
  • 09:23 root@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker2006.codfw.wmnet
  • 09:23 root@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker2006.codfw.wmnet
  • 09:23 root@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker2005.codfw.wmnet
  • 09:23 root@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker2005.codfw.wmnet
  • 09:22 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-web1001.eqiad.wmnet
  • 09:21 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast4006.wikimedia.org
  • 09:19 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2027.codfw.wmnet
  • 09:19 root@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker2005.codfw.wmnet
  • 09:18 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
  • 09:18 root@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker2005.codfw.wmnet
  • 09:18 root@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker2004.codfw.wmnet
  • 09:18 root@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker2004.codfw.wmnet
  • 09:16 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host an-web1001.eqiad.wmnet
  • 09:15 btullis@cumin1004: START - Cookbook sre.ceph.roll-restart-reboot-server rolling reboot on A:cephosd-eqiad
  • 09:14 root@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker2004.codfw.wmnet
  • 09:13 root@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker2004.codfw.wmnet
  • 09:13 root@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker2003.codfw.wmnet
  • 09:13 root@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker2003.codfw.wmnet
  • 09:10 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6004.drmrs.wmnet
  • 09:10 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6004.drmrs.wmnet
  • 09:10 btullis@cumin1004: END (PASS) - Cookbook sre.ceph.roll-restart-reboot-server (exit_code=0) rolling reboot on A:cephosd-codfw
  • 09:09 root@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker2003.codfw.wmnet
  • 09:09 root@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker2003.codfw.wmnet
  • 09:08 root@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker2002.codfw.wmnet
  • 09:08 root@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker2002.codfw.wmnet
  • 09:08 herron@cumin1004: END (PASS) - Cookbook sre.opensearch.roll-restart-reboot (exit_code=0) rolling reboot on A:logstash-codfw
  • 09:04 root@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker2002.codfw.wmnet
  • 09:04 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti6004.drmrs.wmnet
  • 09:04 root@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker2002.codfw.wmnet
  • 09:04 root@cumin2003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-codfw
  • 09:03 klausman@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-lab1002.eqiad.wmnet
  • 09:00 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast4006.wikimedia.org
  • 08:58 slyngshede@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp7005.magru.wmnet
  • 08:58 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P{lvs3010.*} and A:liberica
  • 08:57 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6004.drmrs.wmnet
  • 08:57 klausman@cumin1004: START - Cookbook sre.hosts.reboot-single for host ml-lab1002.eqiad.wmnet
  • 08:55 klausman@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-build1001.eqiad.wmnet
  • 08:54 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin rebooting P{lvs3010.*} and A:liberica
  • 08:53 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P{lvs4010.*,lvs5006.*,lvs6003.*} and A:liberica
  • 08:52 jiji@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc2002.codfw.wmnet
  • 08:51 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host install1005.wikimedia.org
  • 08:50 klausman@cumin1004: START - Cookbook sre.hosts.reboot-single for host ml-build1001.eqiad.wmnet
  • 08:50 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5004.eqsin.wmnet
  • 08:50 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5004.eqsin.wmnet
  • 08:49 trueg@deploy1003: helmfile [eqiad] DONE helmfile.d/services/eventgate-analytics: sync
  • 08:49 trueg@deploy1003: helmfile [eqiad] START helmfile.d/services/eventgate-analytics: sync
  • 08:45 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host install1005.wikimedia.org
  • 08:44 moritzm: failover Ganeti master in drms02 to ganeti6002
  • 08:44 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host install2005.wikimedia.org
  • 08:44 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti5004.eqsin.wmnet
  • 08:43 jiji@cumin1004: START - Cookbook sre.hosts.reboot-single for host mc-misc2002.codfw.wmnet
  • 08:41 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin rebooting P{lvs4010.*,lvs5006.*,lvs6003.*} and A:liberica
  • 08:39 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6002.drmrs.wmnet
  • 08:39 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6002.drmrs.wmnet
  • 08:37 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host install2005.wikimedia.org
  • 08:37 jiji@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc2001.codfw.wmnet
  • 08:35 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host install3004.wikimedia.org
  • 08:33 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti6002.drmrs.wmnet
  • 08:33 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5004.eqsin.wmnet
  • 08:31 jiji@cumin1004: START - Cookbook sre.hosts.reboot-single for host mc-misc2001.codfw.wmnet
  • 08:29 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-coord1004.eqiad.wmnet
  • 08:29 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host install3004.wikimedia.org
  • 08:28 btullis@cumin1004: START - Cookbook sre.ceph.roll-restart-reboot-server rolling reboot on A:cephosd-codfw
  • 08:25 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host install4004.wikimedia.org
  • 08:24 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6002.drmrs.wmnet
  • 08:23 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host an-coord1004.eqiad.wmnet
  • 08:22 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6003.drmrs.wmnet
  • 08:22 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6003.drmrs.wmnet
  • 08:22 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-restart-varnish (exit_code=0) rolling restart of Varnish on 1 hosts matching query P{cp3071.*}
  • 08:20 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-restart-varnish rolling restart of Varnish on 1 hosts matching query P{cp3071.*}
  • 08:19 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host install4004.wikimedia.org
  • 08:18 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'llmisvc' .
  • 08:17 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
  • 08:16 slyngshede@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp7004.magru.wmnet
  • 08:16 moritzm: failover Ganeti master in eqsin to ganeti5007
  • 08:16 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti6003.drmrs.wmnet
  • 08:11 klausman@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for ml-serve1015.eqiad.wmnet
  • 08:11 klausman@cumin1004: START - Cookbook sre.hosts.remove-downtime for ml-serve1015.eqiad.wmnet
  • 08:11 trueg@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply
  • 08:10 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5005.eqsin.wmnet
  • 08:10 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5005.eqsin.wmnet
  • 08:09 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host install5004.wikimedia.org
  • 08:06 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-mariadb1002.eqiad.wmnet
  • 08:06 trueg@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply
  • 08:04 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6003.drmrs.wmnet
  • 08:04 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti5005.eqsin.wmnet
  • 08:03 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
  • 08:02 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
  • 08:02 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
  • 08:00 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host install5004.wikimedia.org
  • 08:00 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-master1004.eqiad.wmnet
  • 07:59 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host an-mariadb1002.eqiad.wmnet
  • 07:56 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host an-master1004.eqiad.wmnet
  • 07:56 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
  • 07:52 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5005.eqsin.wmnet
  • 07:51 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet
  • 07:51 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5006.eqsin.wmnet
  • 07:46 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host install6003.wikimedia.org
  • 07:45 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host install6003.wikimedia.org
  • 07:44 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti5006.eqsin.wmnet
  • 07:40 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
  • 07:36 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
  • 07:34 slyngshede@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp7003.magru.wmnet
  • 07:32 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
  • 07:20 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
  • 07:17 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
  • 07:07 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host install7002.wikimedia.org
  • 07:01 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host install7002.wikimedia.org
  • 06:59 moritzm: failover Ganeti master in drms01 to ganeti6001
  • 06:59 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
  • 06:57 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host releases1003.eqiad.wmnet
  • 06:52 slyngshede@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp7001.magru.wmnet
  • 06:51 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host releases1003.eqiad.wmnet
  • 06:49 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6001.drmrs.wmnet
  • 06:49 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet
  • 06:49 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6001.drmrs.wmnet
  • 06:48 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5007.eqsin.wmnet
  • 06:47 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host zuul1004.eqiad.wmnet
  • 06:47 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
  • 06:43 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti6001.drmrs.wmnet
  • 06:41 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host zuul1004.eqiad.wmnet
  • 06:41 slyngshede@cumin1004: START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-text_magru and not P{cp7002*} and not P{cp7010*} and A:cp
  • 06:40 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti5007.eqsin.wmnet
  • 06:40 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gitlab1004.wikimedia.org
  • 06:37 arnaudb@cumin1004: END (FAIL) - Cookbook sre.gerrit.reboot-gerrit (exit_code=99) Rebooting Gerrit on gerrit1003
  • 06:37 arnaudb@cumin1004: START - Cookbook sre.gerrit.reboot-gerrit Rebooting Gerrit on gerrit1003
  • 06:37 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host phab1005.eqiad.wmnet
  • 06:33 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aphlict1002.eqiad.wmnet
  • 06:32 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
  • 06:31 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6001.drmrs.wmnet
  • 06:30 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host gitlab1004.wikimedia.org
  • 06:30 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet
  • 06:30 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host phab1005.eqiad.wmnet
  • 06:29 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host aphlict1002.eqiad.wmnet
  • 06:10 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host releases2003.codfw.wmnet
  • 06:07 arnaudb@dns1006: END - running authdns-update
  • 06:06 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host releases2003.codfw.wmnet
  • 06:04 arnaudb@dns1006: START - running authdns-update
  • 06:03 arnaudb: moving gitlab behind CDN T425441
  • 05:45 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.decommission (exit_code=0)
  • 05:45 marostegui@cumin1004: Removing db1156 from zarcillo T441298
  • 05:44 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db1156.eqiad.wmnet
  • 05:44 marostegui@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 05:44 marostegui@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db1156.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1004"
  • 05:44 marostegui@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db1156.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1004"
  • 05:39 marostegui@cumin1004: START - Cookbook sre.dns.netbox
  • 05:33 marostegui@cumin1004: START - Cookbook sre.hosts.decommission for hosts db1156.eqiad.wmnet
  • 05:33 marostegui@cumin1004: START - Cookbook sre.mysql.decommission
  • 05:30 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1160.eqiad.wmnet with reason: Maintenance
  • 04:01 mwpresync@deploy1003: Pruned MediaWiki: 1.47.0-wmf.22 (duration: 01m 01s)
  • 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 37s)
  • 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image
  • 00:17 dwisehaupt@dns1004: END - running authdns-update
  • 00:14 dwisehaupt@dns1004: START - running authdns-update
  • 00:13 jdlrobson@deploy1003: Finished scap sync-world: Backport for Drop MediaViewerMobileBeta (T436799) (duration: 10m 50s)
  • 00:08 jdlrobson@deploy1003: jdlrobson: Continuing with deployment
  • 00:07 jdlrobson@deploy1003: jdlrobson: Backport for Drop MediaViewerMobileBeta (T436799) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 00:03 jdlrobson@deploy1003: Started scap sync-world: Backport for Drop MediaViewerMobileBeta (T436799)
  • 00:00 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
  • 00:00 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
  • 00:00 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
  • 00:00 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: sync

2026-10-19

  • 23:32 dwisehaupt@dns1004: END - running authdns-update
  • 23:30 dwisehaupt@dns1004: START - running authdns-update
  • 23:29 sbassett: Deployed security fix for T441038
  • 22:42 sbassett: Deployed security fix for T441639
  • 22:27 sbassett: Deployed security fix for T441632
  • 22:08 sbassett: Deployed security fix for T440946
  • 21:50 maryum: Deployed security fix for T440498
  • 21:47 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host flink-zk2003.codfw.wmnet
  • 21:43 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host flink-zk2003.codfw.wmnet
  • 21:40 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host flink-zk2002.codfw.wmnet
  • 21:39 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host krb2002.codfw.wmnet
  • 21:36 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host flink-zk2002.codfw.wmnet
  • 21:36 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host flink-zk2001.codfw.wmnet
  • 21:33 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host krb2002.codfw.wmnet
  • 21:32 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host flink-zk2001.codfw.wmnet
  • 21:31 maryum: Deployed security fix for T441248
  • 21:30 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-ctrl2002.codfw.wmnet
  • 21:27 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host flink-zk1003.eqiad.wmnet
  • 21:26 dwisehaupt@dns1005: END - running authdns-update
  • 21:24 dwisehaupt@dns1005: START - running authdns-update
  • 21:23 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host flink-zk1003.eqiad.wmnet
  • 21:23 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-ctrl2002.codfw.wmnet
  • 21:18 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host flink-zk1002.eqiad.wmnet
  • 21:14 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-ctrl1002.eqiad.wmnet
  • 21:14 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host flink-zk1002.eqiad.wmnet
  • 21:12 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host flink-zk1001.eqiad.wmnet
  • 21:09 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-ctrl1002.eqiad.wmnet
  • 21:09 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-ctrl2001.codfw.wmnet
  • 21:08 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host flink-zk1001.eqiad.wmnet
  • 21:06 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-etcd2003.codfw.wmnet
  • 21:04 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-ctrl2001.codfw.wmnet
  • 21:04 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-ctrl1001.eqiad.wmnet
  • 21:02 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T441682, restore commons categories lost due to the empty 20261017 weekly dump) xfer categories from wdqs2007.codfw.wmnet -> wdqs2008.codfw.wmnet, repooling both afterwards
  • 21:02 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-etcd2003.codfw.wmnet
  • 21:00 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-etcd2002.codfw.wmnet
  • 21:00 bking@cumin2003: END (ERROR) - Cookbook sre.wdqs.data-transfer (exit_code=97) (T441682, restore commons categories lost due to the empty 20261017 weekly dump) xfer categories from wdqs2007.codfw.wmnet -> wdqs2008.codfw.wmnet, repooling both afterwards
  • 21:00 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer (T441682, restore commons categories lost due to the empty 20261017 weekly dump) xfer categories from wdqs2007.codfw.wmnet -> wdqs2008.codfw.wmnet, repooling both afterwards
  • 20:59 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-ctrl1001.eqiad.wmnet
  • 20:56 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-etcd2002.codfw.wmnet
  • 20:56 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-etcd2001.codfw.wmnet
  • 20:55 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer (T441682, restore commons categories lost due to the empty 20261017 weekly dump) xfer categories from wdqs2007.codfw.wmnet -> wdqs2008.codfw.wmnet, repooling both afterwards
  • 20:54 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer (T441682, restore commons categories lost due to the empty 20261017 weekly dump) xfer categories from wdqs2007.codfw.wmnet -> wdqs2008.codfw.wmnet, repooling both afterwards
  • 20:54 btullis@cumin1004: END (PASS) - Cookbook sre.misc-clusters.roll-restart-reboot-eventschemas (exit_code=0) rolling reboot on A:schema
  • 20:52 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-etcd2001.codfw.wmnet
  • 20:51 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-etcd1003.eqiad.wmnet
  • 20:51 dancy@deploy1003: Installation of scap version "4.301.0" completed for 3 hosts
  • 20:49 dancy@deploy1003: Installing scap version "4.301.0" for 3 host(s)
  • 20:47 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-etcd1003.eqiad.wmnet
  • 20:47 dwisehaupt@dns1005: END - running authdns-update
  • 20:45 dwisehaupt@dns1005: START - running authdns-update
  • 20:44 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply
  • 20:43 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-etcd1002.eqiad.wmnet
  • 20:43 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply
  • 20:40 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-etcd1002.eqiad.wmnet
  • 20:39 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-etcd1001.eqiad.wmnet
  • 20:37 btullis@cumin1004: START - Cookbook sre.misc-clusters.roll-restart-reboot-eventschemas rolling reboot on A:schema
  • 20:35 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-etcd1001.eqiad.wmnet
  • 20:34 krinkle@deploy1003: Finished scap sync-world: Backport for OutputPageHooks: Avoid DB query for talk page exists() on every pageview (T347123) (1.47.0-wmf.24) (duration: 12m 07s)
  • 20:28 krinkle@deploy1003: krinkle: Continuing with deployment
  • 20:28 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply
  • 20:27 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply
  • 20:26 krinkle@deploy1003: krinkle: Backport for OutputPageHooks: Avoid DB query for talk page exists() on every pageview (T347123) (1.47.0-wmf.24) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 20:22 krinkle@deploy1003: Started scap sync-world: Backport for OutputPageHooks: Avoid DB query for talk page exists() on every pageview (T347123) (1.47.0-wmf.24)
  • 20:17 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply
  • 20:17 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply
  • 20:15 arlolra@deploy1003: Finished scap sync-world: Backport for Ignore lints when parsing non-lintable content models (T441307) (1.47.0-wmf.24) (duration: 09m 59s)
  • 20:10 arlolra@deploy1003: arlolra: Continuing with deployment
  • 20:09 arlolra@deploy1003: arlolra: Backport for Ignore lints when parsing non-lintable content models (T441307) (1.47.0-wmf.24) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 20:05 arlolra@deploy1003: Started scap sync-world: Backport for Ignore lints when parsing non-lintable content models (T441307) (1.47.0-wmf.24)
  • 19:20 ssastry@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
  • 19:19 ssastry@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
  • 19:19 ssastry@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
  • 19:19 ssastry@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
  • 19:06 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
  • 18:59 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
  • 17:56 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1221 (T441197)', diff saved to https://phabricator.wikimedia.org/P96857 and previous config saved to /var/cache/conftool/dbconfig/20261019-175642-marostegui.json
  • 17:43 krinkle@deploy1003: Finished scap sync-world: Backport for GlobalFunctions: Remove Microsoft IIS 7.5 workaround from wfUrlencode (T441517) (1.47.0-wmf.24), VisualEditorResourceModule: Remove unused Message::exists() check (T441253) (1.47.0-wmf.24) (duration: 11m 49s)
  • 17:41 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1221', diff saved to https://phabricator.wikimedia.org/P96856 and previous config saved to /var/cache/conftool/dbconfig/20261019-174140-marostegui.json
  • 17:37 krinkle@deploy1003: krinkle: Continuing with deployment
  • 17:35 krinkle@deploy1003: krinkle: Backport for GlobalFunctions: Remove Microsoft IIS 7.5 workaround from wfUrlencode (T441517) (1.47.0-wmf.24), VisualEditorResourceModule: Remove unused Message::exists() check (T441253) (1.47.0-wmf.24) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 17:31 krinkle@deploy1003: Started scap sync-world: Backport for GlobalFunctions: Remove Microsoft IIS 7.5 workaround from wfUrlencode (T441517) (1.47.0-wmf.24), VisualEditorResourceModule: Remove unused Message::exists() check (T441253) (1.47.0-wmf.24)
  • 17:31 herron@deploy1003: Finished scap sync-world: bump rsyslog to 8.2608.0-4 T438375 (duration: 10m 45s)
  • 17:28 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2223 (T441633)', diff saved to https://phabricator.wikimedia.org/P96855 and previous config saved to /var/cache/conftool/dbconfig/20261019-172847-ladsgroup.json
  • 17:28 btullis@cumin1004: END (PASS) - Cookbook sre.druid.reboot-workers (exit_code=0) for Druid public cluster: Reboot Druid nodes
  • 17:28 mutante: sre.host.reboot-single fails entirely if the linked task isn't public so it cant update it
  • 17:27 dzahn@cumin1004: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host zuul1005.eqiad.wmnet
  • 17:27 dzahn@cumin1004: START - Cookbook sre.hosts.reboot-single for host zuul1005.eqiad.wmnet
  • 17:27 cwilliams@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host backup2020.codfw.wmnet
  • 17:26 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1221', diff saved to https://phabricator.wikimedia.org/P96854 and previous config saved to /var/cache/conftool/dbconfig/20261019-172638-marostegui.json
  • 17:25 cwilliams@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host backup1020.eqiad.wmnet
  • 17:24 cwilliams@cumin1004: START - Cookbook sre.hosts.reboot-single for host backup2020.codfw.wmnet
  • 17:23 dzahn@cumin1004: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host zuul1004.eqiad.wmnet
  • 17:23 dzahn@cumin1004: START - Cookbook sre.hosts.reboot-single for host zuul1004.eqiad.wmnet
  • 17:22 cwilliams@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host backup2019.codfw.wmnet
  • 17:22 cwilliams@cumin1004: START - Cookbook sre.hosts.reboot-single for host backup1020.eqiad.wmnet
  • 17:22 herron@deploy1003: Started scap sync-world: bump rsyslog to 8.2608.0-4 T438375
  • 17:21 cwilliams@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host backup1019.eqiad.wmnet
  • 17:20 cwilliams@cumin1004: START - Cookbook sre.hosts.reboot-single for host backup2019.codfw.wmnet
  • 17:18 cwilliams@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host backup2018.codfw.wmnet
  • 17:18 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2223', diff saved to https://phabricator.wikimedia.org/P96853 and previous config saved to /var/cache/conftool/dbconfig/20261019-171846-ladsgroup.json
  • 17:18 cwilliams@cumin1004: START - Cookbook sre.hosts.reboot-single for host backup1019.eqiad.wmnet
  • 17:17 cwilliams@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host backup1018.eqiad.wmnet
  • 17:15 cwilliams@cumin1004: START - Cookbook sre.hosts.reboot-single for host backup2018.codfw.wmnet
  • 17:14 cwilliams@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host backup2017.codfw.wmnet
  • 17:14 cwilliams@cumin1004: START - Cookbook sre.hosts.reboot-single for host backup1018.eqiad.wmnet
  • 17:13 cwilliams@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host backup1017.eqiad.wmnet
  • 17:11 cwilliams@cumin1004: START - Cookbook sre.hosts.reboot-single for host backup2017.codfw.wmnet
  • 17:11 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1221 (T441197)', diff saved to https://phabricator.wikimedia.org/P96852 and previous config saved to /var/cache/conftool/dbconfig/20261019-171136-marostegui.json
  • 17:10 cwilliams@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host backup2016.codfw.wmnet
  • 17:10 cwilliams@cumin1004: START - Cookbook sre.hosts.reboot-single for host backup1017.eqiad.wmnet
  • 17:09 cwilliams@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host backup1016.eqiad.wmnet
  • 17:08 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2223', diff saved to https://phabricator.wikimedia.org/P96851 and previous config saved to /var/cache/conftool/dbconfig/20261019-170844-ladsgroup.json
  • 17:07 cwilliams@cumin1004: START - Cookbook sre.hosts.reboot-single for host backup2016.codfw.wmnet
  • 17:06 cwilliams@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host backup2015.codfw.wmnet
  • 17:06 cwilliams@cumin1004: START - Cookbook sre.hosts.reboot-single for host backup1016.eqiad.wmnet
  • 17:04 cwilliams@cumin1004: START - Cookbook sre.hosts.reboot-single for host backup2015.codfw.wmnet
  • 17:03 marostegui@cumin1004: dbctl commit (dc=all): 'Depooling db1221 (T441197)', diff saved to https://phabricator.wikimedia.org/P96850 and previous config saved to /var/cache/conftool/dbconfig/20261019-170345-marostegui.json
  • 17:03 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on 6 hosts with reason: Maintenance
  • 17:03 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1221.eqiad.wmnet with reason: Maintenance
  • 16:58 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2223 (T441633)', diff saved to https://phabricator.wikimedia.org/P96849 and previous config saved to /var/cache/conftool/dbconfig/20261019-165842-ladsgroup.json
  • 16:52 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2223 (T441633)', diff saved to https://phabricator.wikimedia.org/P96848 and previous config saved to /var/cache/conftool/dbconfig/20261019-165214-ladsgroup.json
  • 16:52 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2223.codfw.wmnet with reason: Maintenance
  • 16:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2213 (T441633)', diff saved to https://phabricator.wikimedia.org/P96847 and previous config saved to /var/cache/conftool/dbconfig/20261019-165152-ladsgroup.json
  • 16:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P96846 and previous config saved to /var/cache/conftool/dbconfig/20261019-164150-ladsgroup.json
  • 16:38 kamila@deploy1003: Finished scap deploy-service shellbox: roll out PHP 8.5 for -timeline and -video (duration: 04m 57s)
  • 16:35 kamila@deploy1003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
  • 16:33 kamila@deploy1003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] codfw Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
  • 16:33 kamila@deploy1003: scap deploy-service shellbox: [STAGING/DEFAULT] staging-eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
  • 16:33 kamila@deploy1003: Started scap deploy-service shellbox: roll out PHP 8.5 for -timeline and -video
  • 16:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P96845 and previous config saved to /var/cache/conftool/dbconfig/20261019-163148-ladsgroup.json
  • 16:28 cwilliams@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host backup1015.eqiad.wmnet
  • 16:26 herron@cumin1004: START - Cookbook sre.opensearch.roll-restart-reboot rolling reboot on A:logstash-codfw
  • 16:25 cwilliams@cumin1004: START - Cookbook sre.hosts.reboot-single for host backup1015.eqiad.wmnet
  • 16:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2213 (T441633)', diff saved to https://phabricator.wikimedia.org/P96844 and previous config saved to /var/cache/conftool/dbconfig/20261019-162147-ladsgroup.json
  • 16:21 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
  • 16:17 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1265.eqiad.wmnet with reason: Maintenance
  • 16:17 cwilliams@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-backup1004.eqiad.wmnet
  • 16:17 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1252 (T441197)', diff saved to https://phabricator.wikimedia.org/P96843 and previous config saved to /var/cache/conftool/dbconfig/20261019-161725-marostegui.json
  • 16:16 cwilliams@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-backup2004.codfw.wmnet
  • 16:16 btullis@cumin1004: START - Cookbook sre.druid.reboot-workers for Druid public cluster: Reboot Druid nodes
  • 16:14 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2213 (T441633)', diff saved to https://phabricator.wikimedia.org/P96842 and previous config saved to /var/cache/conftool/dbconfig/20261019-161456-ladsgroup.json
  • 16:14 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2213.codfw.wmnet with reason: Maintenance
  • 16:14 cwilliams@cumin1004: START - Cookbook sre.hosts.reboot-single for host ms-backup1004.eqiad.wmnet
  • 16:14 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2211 (T441633)', diff saved to https://phabricator.wikimedia.org/P96841 and previous config saved to /var/cache/conftool/dbconfig/20261019-161435-ladsgroup.json
  • 16:13 cwilliams@cumin1004: START - Cookbook sre.hosts.reboot-single for host ms-backup2004.codfw.wmnet
  • 16:11 btullis@cumin1004: END (PASS) - Cookbook sre.druid.reboot-workers (exit_code=0) for Druid analytics cluster: Reboot Druid nodes
  • 16:05 cwilliams@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-backup2003.codfw.wmnet
  • 16:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2211', diff saved to https://phabricator.wikimedia.org/P96840 and previous config saved to /var/cache/conftool/dbconfig/20261019-160433-ladsgroup.json
  • 16:03 cwilliams@cumin1004: START - Cookbook sre.hosts.reboot-single for host ms-backup2003.codfw.wmnet
  • 16:02 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1252', diff saved to https://phabricator.wikimedia.org/P96839 and previous config saved to /var/cache/conftool/dbconfig/20261019-160224-marostegui.json
  • 15:59 cwilliams@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-backup1003.eqiad.wmnet
  • 15:56 cwilliams@cumin1004: START - Cookbook sre.hosts.reboot-single for host ms-backup1003.eqiad.wmnet
  • 15:56 jdrewniak@deploy1003: Finished scap sync-world: Backport for Bumping portals to master (T128546) (duration: 12m 54s)
  • 15:55 cwilliams@cumin1004: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host ms-backup1003.eqiad.wmnet
  • 15:55 cwilliams@cumin1004: START - Cookbook sre.hosts.reboot-single for host ms-backup1003.eqiad.wmnet
  • 15:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2211', diff saved to https://phabricator.wikimedia.org/P96838 and previous config saved to /var/cache/conftool/dbconfig/20261019-155431-ladsgroup.json
  • 15:50 jdrewniak@deploy1003: jdrewniak: Continuing with deployment
  • 15:47 jdrewniak@deploy1003: jdrewniak: Backport for Bumping portals to master (T128546) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 15:47 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1252', diff saved to https://phabricator.wikimedia.org/P96837 and previous config saved to /var/cache/conftool/dbconfig/20261019-154721-marostegui.json
  • 15:46 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.categories-reload (exit_code=0) reloading categories to wdqs2008.codfw.wmnet
  • 15:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2211 (T441633)', diff saved to https://phabricator.wikimedia.org/P96836 and previous config saved to /var/cache/conftool/dbconfig/20261019-154429-ladsgroup.json
  • 15:43 jdrewniak@deploy1003: Started scap sync-world: Backport for Bumping portals to master (T128546)
  • 15:37 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2211 (T441633)', diff saved to https://phabricator.wikimedia.org/P96835 and previous config saved to /var/cache/conftool/dbconfig/20261019-153735-ladsgroup.json
  • 15:37 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2211.codfw.wmnet with reason: Maintenance
  • 15:36 dancy@deploy1003: Installation of scap version "4.300.0" completed for 3 hosts
  • 15:36 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-maint2001.codfw.wmnet
  • 15:34 dancy@deploy1003: Installing scap version "4.300.0" for 3 host(s)
  • 15:34 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
  • 15:33 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'.
  • 15:32 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ldap-maint2001.codfw.wmnet
  • 15:32 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1252 (T441197)', diff saved to https://phabricator.wikimedia.org/P96834 and previous config saved to /var/cache/conftool/dbconfig/20261019-153219-marostegui.json
  • 15:32 mpostoronca@deploy1003: Finished scap sync-world: Backport for SECURITY: AbuseReview: Respect hidden usernames in username filter (T439752) (1.47.0-wmf.24) (duration: 12m 12s)
  • 15:32 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2201.codfw.wmnet with reason: Maintenance
  • 15:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2192 (T441633)', diff saved to https://phabricator.wikimedia.org/P96833 and previous config saved to /var/cache/conftool/dbconfig/20261019-153155-ladsgroup.json
  • 15:31 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'.
  • 15:31 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-maint1001.eqiad.wmnet
  • 15:30 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'.
  • 15:29 cgoubert@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
  • 15:28 cgoubert@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
  • 15:28 cgoubert@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'.
  • 15:28 cgoubert@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'.
  • 15:27 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ldap-maint1001.eqiad.wmnet
  • 15:27 marostegui@dns1004: END - running authdns-update
  • 15:26 mpostoronca@deploy1003: mpostoronca: Continuing with deployment
  • 15:26 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply
  • 15:26 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-video: apply
  • 15:26 jmm@cumin2003: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti4005.ulsfo.wmnet
  • 15:26 jmm@cumin2003: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host ganeti4005.ulsfo.wmnet
  • 15:26 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply
  • 15:25 marostegui@dns1004: START - running authdns-update
  • 15:24 mpostoronca@deploy1003: mpostoronca: Backport for SECURITY: AbuseReview: Respect hidden usernames in username filter (T439752) (1.47.0-wmf.24) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 15:23 marostegui@cumin1004: dbctl commit (dc=all): 'Depooling db1252 (T441197)', diff saved to https://phabricator.wikimedia.org/P96832 and previous config saved to /var/cache/conftool/dbconfig/20261019-152315-marostegui.json
  • 15:23 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1252.eqiad.wmnet with reason: Maintenance
  • 15:23 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1249 (T441197)', diff saved to https://phabricator.wikimedia.org/P96831 and previous config saved to /var/cache/conftool/dbconfig/20261019-152257-marostegui.json
  • 15:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2192', diff saved to https://phabricator.wikimedia.org/P96830 and previous config saved to /var/cache/conftool/dbconfig/20261019-152153-ladsgroup.json
  • 15:20 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-video: apply
  • 15:20 mpostoronca@deploy1003: Started scap sync-world: Backport for SECURITY: AbuseReview: Respect hidden usernames in username filter (T439752) (1.47.0-wmf.24)
  • 15:19 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply
  • 15:18 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host zuul1007.eqiad.wmnet
  • 15:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netflow1004.eqiad.wmnet
  • 15:14 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host zuul1007.eqiad.wmnet
  • 15:13 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host zuul1006.eqiad.wmnet
  • 15:13 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'llmisvc' .
  • 15:13 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
  • 15:11 elukey@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'.
  • 15:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2192', diff saved to https://phabricator.wikimedia.org/P96828 and previous config saved to /var/cache/conftool/dbconfig/20261019-151151-ladsgroup.json
  • 15:11 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host build2002.codfw.wmnet
  • 15:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/analytics-test: apply
  • 15:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/analytics-test: apply
  • 15:10 elukey@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'.
  • 15:09 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netflow1004.eqiad.wmnet
  • 15:09 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-video: apply
  • 15:08 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-video: apply
  • 15:08 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netflow1003.eqiad.wmnet
  • 15:08 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host zuul1006.eqiad.wmnet
  • 15:08 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gerrit2003.wikimedia.org
  • 15:08 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-video: apply
  • 15:07 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1249', diff saved to https://phabricator.wikimedia.org/P96827 and previous config saved to /var/cache/conftool/dbconfig/20261019-150755-marostegui.json
  • 15:07 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-video: apply
  • 15:06 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-video: apply
  • 15:06 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-timeline: apply
  • 15:05 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-timeline: apply
  • 15:05 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: apply
  • 15:05 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host build2002.codfw.wmnet
  • 15:04 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: apply
  • 15:04 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: apply
  • 15:04 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti4005.ulsfo.wmnet
  • 15:04 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: apply
  • 15:03 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-syntaxhighlight: apply
  • 15:03 samtar@deploy1003: Finished scap sync-world: Backport for IS: Enable PersonalDashboard on test2wiki (duration: 12m 16s)
  • 15:02 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-syntaxhighlight: apply
  • 15:02 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply
  • 15:01 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2192 (T441633)', diff saved to https://phabricator.wikimedia.org/P96826 and previous config saved to /var/cache/conftool/dbconfig/20261019-150149-ladsgroup.json
  • 15:01 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply
  • 15:01 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host gerrit2003.wikimedia.org
  • 15:01 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gitlab2002.wikimedia.org
  • 15:01 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netflow1003.eqiad.wmnet
  • 15:00 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti4005.ulsfo.wmnet
  • 14:59 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-syntaxhighlight: apply
  • 14:59 btullis@cumin1004: START - Cookbook sre.druid.reboot-workers for Druid analytics cluster: Reboot Druid nodes
  • 14:58 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-syntaxhighlight: apply
  • 14:58 btullis@cumin1004: END (PASS) - Cookbook sre.presto.reboot-workers (exit_code=0) for Presto an-presto cluster: Reboot Presto nodes
  • 14:58 btullis@cumin1004: START - Cookbook sre.hadoop.reboot-workers for Hadoop analytics cluster
  • 14:57 samtar@deploy1003: samtar: Continuing with deployment
  • 14:56 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-media: apply
  • 14:55 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-media: apply
  • 14:55 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-media: apply
  • 14:55 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2192 (T441633)', diff saved to https://phabricator.wikimedia.org/P96825 and previous config saved to /var/cache/conftool/dbconfig/20261019-145507-ladsgroup.json
  • 14:55 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2192.codfw.wmnet with reason: Maintenance
  • 14:55 samtar@deploy1003: samtar: Backport for IS: Enable PersonalDashboard on test2wiki synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 14:55 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-media: apply
  • 14:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2178 (T441633)', diff saved to https://phabricator.wikimedia.org/P96824 and previous config saved to /var/cache/conftool/dbconfig/20261019-145445-ladsgroup.json
  • 14:54 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host gitlab2002.wikimedia.org
  • 14:54 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gitlab1003.wikimedia.org
  • 14:53 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-media: apply
  • 14:53 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-media: apply
  • 14:52 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1249', diff saved to https://phabricator.wikimedia.org/P96823 and previous config saved to /var/cache/conftool/dbconfig/20261019-145253-marostegui.json
  • 14:51 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-constraints: apply
  • 14:51 samtar@deploy1003: Started scap sync-world: Backport for IS: Enable PersonalDashboard on test2wiki
  • 14:50 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-constraints: apply
  • 14:50 sukhe@cumin1004: END (PASS) - Cookbook sre.cdn.roll-restart-reboot-hcaptcha-proxy (exit_code=0) rolling reboot on A:hcaptcha-proxy and A:hcaptcha-proxy
  • 14:47 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host gitlab1003.wikimedia.org
  • 14:46 sukhe@cumin1004: END (PASS) - Cookbook sre.dns.roll-restart-reboot-durum (exit_code=0) rolling reboot on A:durum
  • 14:46 sukhe@cumin1004: END (PASS) - Cookbook sre.dns.roll-restart-reboot-wikimedia-dns (exit_code=0) rolling reboot on A:wikidough
  • 14:46 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-constraints: apply
  • 14:45 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-constraints: apply
  • 14:45 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-constraints: apply
  • 14:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2178', diff saved to https://phabricator.wikimedia.org/P96822 and previous config saved to /var/cache/conftool/dbconfig/20261019-144443-ladsgroup.json
  • 14:44 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-constraints: apply
  • 14:44 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) upgrade A:liberica-esams
  • 14:44 ladsgroup@deploy1003: Finished scap sync-world: Backport for hCaptcha: Remove ResourceLoader dependency from HCaptchaOutput (T441681) (1.47.0-wmf.24) (duration: 11m 50s)
  • 14:43 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox: apply
  • 14:43 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox: apply
  • 14:42 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox: apply
  • 14:42 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.upgrade upgrade A:liberica-esams
  • 14:42 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host tcp-proxy7002.magru.wmnet
  • 14:41 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) upgrade A:liberica-drmrs
  • 14:41 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-conf1006.eqiad.wmnet
  • 14:41 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox: apply
  • 14:41 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: apply
  • 14:40 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: apply
  • 14:39 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.upgrade upgrade A:liberica-drmrs
  • 14:39 ladsgroup@deploy1003: ladsgroup: Continuing with deployment
  • 14:38 ladsgroup@deploy1003: ladsgroup: Backport for hCaptcha: Remove ResourceLoader dependency from HCaptchaOutput (T441681) (1.47.0-wmf.24) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 14:37 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1249 (T441197)', diff saved to and previous config saved to /var/cache/conftool/dbconfig/20261019-143751-marostegui.json
  • 14:37 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host tcp-proxy7002.magru.wmnet
  • 14:37 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host tcp-proxy7001.magru.wmnet
  • 14:37 sukhe@cumin1004: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P{cp7010* or cp7002*} and A:cp
  • 14:37 sukhe@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp7010.magru.wmnet
  • 14:36 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host an-conf1006.eqiad.wmnet
  • 14:36 bking@cumin2003: START - Cookbook sre.wdqs.categories-reload reloading categories to wdqs2008.codfw.wmnet
  • 14:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2178', diff saved to https://phabricator.wikimedia.org/P96821 and previous config saved to /var/cache/conftool/dbconfig/20261019-143442-ladsgroup.json
  • 14:32 ladsgroup@deploy1003: Started scap sync-world: Backport for hCaptcha: Remove ResourceLoader dependency from HCaptchaOutput (T441681) (1.47.0-wmf.24)
  • 14:32 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host tcp-proxy7001.magru.wmnet
  • 14:32 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host tcp-proxy6002.drmrs.wmnet
  • 14:30 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host tcp-proxy6002.drmrs.wmnet
  • 14:30 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host tcp-proxy6001.drmrs.wmnet
  • 14:30 marostegui@cumin1004: dbctl commit (dc=all): 'Depooling db1249 (T441197)', diff saved to https://phabricator.wikimedia.org/P96820 and previous config saved to /var/cache/conftool/dbconfig/20261019-143004-marostegui.json
  • 14:30 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1249.eqiad.wmnet with reason: Maintenance
  • 14:29 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1248 (T441197)', diff saved to https://phabricator.wikimedia.org/P96819 and previous config saved to /var/cache/conftool/dbconfig/20261019-142947-marostegui.json
  • 14:29 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) upgrade A:liberica-eqsin
  • 14:27 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-conf1005.eqiad.wmnet
  • 14:26 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.upgrade upgrade A:liberica-eqsin
  • 14:25 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host tcp-proxy6001.drmrs.wmnet
  • 14:25 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host tcp-proxy5004.eqsin.wmnet
  • 14:24 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2178 (T441633)', diff saved to https://phabricator.wikimedia.org/P96818 and previous config saved to /var/cache/conftool/dbconfig/20261019-142440-ladsgroup.json
  • 14:22 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host an-conf1005.eqiad.wmnet
  • 14:20 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host tcp-proxy5004.eqsin.wmnet
  • 14:20 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host tcp-proxy5003.eqsin.wmnet
  • 14:19 kamila@deploy1003: Finished scap deploy-service shellbox: pilot PHP 8.5 on -timeline and -video (duration: 04m 35s)
  • 14:18 kamila@deploy1003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
  • 14:18 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2178 (T441633)', diff saved to https://phabricator.wikimedia.org/P96817 and previous config saved to /var/cache/conftool/dbconfig/20261019-141807-ladsgroup.json
  • 14:18 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2178.codfw.wmnet with reason: Maintenance
  • 14:17 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2171 (T441633)', diff saved to https://phabricator.wikimedia.org/P96816 and previous config saved to /var/cache/conftool/dbconfig/20261019-141756-ladsgroup.json
  • 14:15 kamila@deploy1003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] codfw Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
  • 14:15 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host tcp-proxy5003.eqsin.wmnet
  • 14:15 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host tcp-proxy4004.ulsfo.wmnet
  • 14:15 kamila@deploy1003: Started scap deploy-service shellbox: pilot PHP 8.5 on -timeline and -video
  • 14:15 kamila@deploy1003: scap deploy-service shellbox: [STAGING/DEFAULT] staging-eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
  • 14:14 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1248', diff saved to https://phabricator.wikimedia.org/P96815 and previous config saved to /var/cache/conftool/dbconfig/20261019-141445-marostegui.json
  • 14:13 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on dbproxy1027.eqiad.wmnet with reason: Kernel reboots
  • 14:11 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on dbproxy1028.eqiad.wmnet with reason: Kernel reboots
  • 14:10 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host tcp-proxy4004.ulsfo.wmnet
  • 14:10 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host tcp-proxy4003.ulsfo.wmnet
  • 14:07 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2171', diff saved to https://phabricator.wikimedia.org/P96814 and previous config saved to /var/cache/conftool/dbconfig/20261019-140754-ladsgroup.json
  • 14:06 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on dbproxy1025.eqiad.wmnet with reason: Kernel reboots
  • 14:05 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host tcp-proxy4003.ulsfo.wmnet
  • 14:05 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host tcp-proxy3002.esams.wmnet
  • 14:05 arnaudb@cumin1004: END (PASS) - Cookbook sre.gitlab.reboot-runner (exit_code=0) rolling reboot on P{gitlab-runner2004.codfw.wmnet} and (A:gitlab-runner)
  • 14:04 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for knwikisource: lift IP cap for workshop on 24-October-2026 (T440546), mlwikivoyage: add logo, wordmark and set sitename, timezone (T441173), ruwikisource: add 'A' as NamespaceAliases for author namespace (T439568) (duration: 11m 57s)
  • 14:02 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on dbproxy1022.eqiad.wmnet with reason: Kernel reboots
  • 14:00 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host tcp-proxy3002.esams.wmnet
  • 14:00 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netflow1002.eqiad.wmnet
  • 13:59 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host vrts1003.eqiad.wmnet
  • 13:59 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1248', diff saved to https://phabricator.wikimedia.org/P96813 and previous config saved to /var/cache/conftool/dbconfig/20261019-135943-marostegui.json
  • 13:59 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host lists1004.wikimedia.org
  • 13:58 lucaswerkmeister-wmde@deploy1003: anzx, lucaswerkmeister-wmde: Continuing with deployment
  • 13:58 arnaudb@cumin1004: START - Cookbook sre.gitlab.reboot-runner rolling reboot on P{gitlab-runner2004.codfw.wmnet} and (A:gitlab-runner)
  • 13:58 arnaudb@cumin1004: END (PASS) - Cookbook sre.gitlab.reboot-runner (exit_code=0) rolling reboot on P{gitlab-runner2003.codfw.wmnet} and (A:gitlab-runner)
  • 13:57 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2171', diff saved to https://phabricator.wikimedia.org/P96812 and previous config saved to /var/cache/conftool/dbconfig/20261019-135752-ladsgroup.json
  • 13:56 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netflow1002.eqiad.wmnet
  • 13:56 lucaswerkmeister-wmde@deploy1003: anzx, lucaswerkmeister-wmde: Backport for knwikisource: lift IP cap for workshop on 24-October-2026 (T440546), mlwikivoyage: add logo, wordmark and set sitename, timezone (T441173), ruwikisource: add 'A' as NamespaceAliases for author namespace (T439568) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 13:56 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host stewards1001.eqiad.wmnet
  • 13:56 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 11 hosts with reason: Kernel reboots
  • 13:56 vgutierrez@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host lvs7003.magru.wmnet
  • 13:55 moritzm: failover Ganeti master in ulsfo to ganeti4008
  • 13:55 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'llmisvc' .
  • 13:55 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
  • 13:55 sukhe@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp7002.magru.wmnet
  • 13:54 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host tcp-proxy3001.esams.wmnet
  • 13:53 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host vrts1003.eqiad.wmnet
  • 13:52 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host lists1004.wikimedia.org
  • 13:52 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host stewards1001.eqiad.wmnet
  • 13:52 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for knwikisource: lift IP cap for workshop on 24-October-2026 (T440546), mlwikivoyage: add logo, wordmark and set sitename, timezone (T441173), ruwikisource: add 'A' as NamespaceAliases for author namespace (T439568)
  • 13:51 arnaudb@cumin1004: START - Cookbook sre.gitlab.reboot-runner rolling reboot on P{gitlab-runner2003.codfw.wmnet} and (A:gitlab-runner)
  • 13:51 sukhe@cumin1004: END (PASS) - Cookbook sre.dns.roll-reboot (exit_code=0) rolling reboot on P{dns7001*} and (A:dnsbox)
  • 13:51 sukhe@cumin1004: cookbooks.sre.dns.roll-reboot finished rebooting dns7001.wikimedia.org
  • 13:50 arnaudb@cumin1004: END (PASS) - Cookbook sre.gitlab.reboot-runner (exit_code=0) rolling reboot on P{gitlab-runner2002.codfw.wmnet} and (A:gitlab-runner)
  • 13:50 sukhe@cumin1004: END (PASS) - Cookbook sre.cdn.roll-restart-reboot-ncredir (exit_code=0) rolling reboot on P{ncredir7004*} and A:ncredir
  • 13:49 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host tcp-proxy3001.esams.wmnet
  • 13:48 vgutierrez@cumin1004: START - Cookbook sre.hosts.reboot-single for host lvs7003.magru.wmnet
  • 13:48 vgutierrez@cumin1004: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host lvs7003.magru.wmnet
  • 13:47 vgutierrez@cumin1004: START - Cookbook sre.hosts.reboot-single for host lvs7003.magru.wmnet
  • 13:47 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2171 (T441633)', diff saved to https://phabricator.wikimedia.org/P96811 and previous config saved to /var/cache/conftool/dbconfig/20261019-134750-ladsgroup.json
  • 13:46 sukhe@cumin1004: START - Cookbook sre.cdn.roll-restart-reboot-ncredir rolling reboot on P{ncredir7004*} and A:ncredir
  • 13:44 sukhe@cumin1004: START - Cookbook sre.cdn.roll-restart-reboot-hcaptcha-proxy rolling reboot on A:hcaptcha-proxy and A:hcaptcha-proxy
  • 13:44 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1248 (T441197)', diff saved to https://phabricator.wikimedia.org/P96810 and previous config saved to /var/cache/conftool/dbconfig/20261019-134441-marostegui.json
  • 13:44 seanleong-wmde@deploy1003: Finished scap sync-world: Backport for Config change for opting out cswikiversity from LL suppression. (T441437) (duration: 37m 55s)
  • 13:43 arnaudb@cumin1004: START - Cookbook sre.gitlab.reboot-runner rolling reboot on P{gitlab-runner2002.codfw.wmnet} and (A:gitlab-runner)
  • 13:43 arnaudb@cumin1004: END (PASS) - Cookbook sre.gitlab.reboot-runner (exit_code=0) rolling reboot on P{gitlab-runner1004.eqiad.wmnet} and (A:gitlab-runner)
  • 13:43 slyngshede@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host idp1005.wikimedia.org
  • 13:43 sukhe@cumin1004: START - Cookbook sre.cdn.roll-reboot rolling reboot on P{cp7010* or cp7002*} and A:cp
  • 13:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2171 (T441633)', diff saved to https://phabricator.wikimedia.org/P96809 and previous config saved to /var/cache/conftool/dbconfig/20261019-134105-ladsgroup.json
  • 13:41 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2171.codfw.wmnet with reason: Maintenance
  • 13:41 sukhe@cumin1004: START - Cookbook sre.dns.roll-restart-reboot-wikimedia-dns rolling reboot on A:wikidough
  • 13:40 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) upgrade A:liberica-ulsfo
  • 13:40 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti4006.ulsfo.wmnet
  • 13:40 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti4006.ulsfo.wmnet
  • 13:40 sukhe@cumin1004: START - Cookbook sre.dns.roll-restart-reboot-durum rolling reboot on A:durum
  • 13:40 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netflow2005.codfw.wmnet
  • 13:39 jmm@cumin2003: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti7001.magru.wmnet
  • 13:39 jmm@cumin2003: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host ganeti7001.magru.wmnet
  • 13:39 sukhe@cumin1004: cookbooks.sre.dns.roll-reboot begin reboot of dns7001.wikimedia.org
  • 13:39 sukhe@cumin1004: START - Cookbook sre.dns.roll-reboot rolling reboot on P{dns7001*} and (A:dnsbox)
  • 13:39 slyngshede@cumin1004: START - Cookbook sre.hosts.reboot-single for host idp1005.wikimedia.org
  • 13:38 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.upgrade upgrade A:liberica-ulsfo
  • 13:38 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host tcp-proxy2002.codfw.wmnet
  • 13:35 arnaudb@cumin1004: START - Cookbook sre.gitlab.reboot-runner rolling reboot on P{gitlab-runner1004.eqiad.wmnet} and (A:gitlab-runner)
  • 13:35 arnaudb@cumin1004: END (PASS) - Cookbook sre.gitlab.reboot-runner (exit_code=0) rolling reboot on P{gitlab-runner1003.eqiad.wmnet} and (A:gitlab-runner)
  • 13:34 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netflow2005.codfw.wmnet
  • 13:34 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti4006.ulsfo.wmnet
  • 13:34 marostegui@cumin1004: dbctl commit (dc=all): 'Depooling db1248 (T441197)', diff saved to https://phabricator.wikimedia.org/P96808 and previous config saved to /var/cache/conftool/dbconfig/20261019-133406-marostegui.json
  • 13:34 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1248.eqiad.wmnet with reason: Maintenance
  • 13:33 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1247 (T441197)', diff saved to https://phabricator.wikimedia.org/P96807 and previous config saved to /var/cache/conftool/dbconfig/20261019-133348-marostegui.json
  • 13:33 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netflow2004.codfw.wmnet
  • 13:33 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host tcp-proxy2002.codfw.wmnet
  • 13:33 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host tcp-proxy2002.codfw.wmnet
  • 13:32 slyngshede@dns1004: END - running authdns-update
  • 13:32 seanleong-wmde@deploy1003: seanleong-wmde: Continuing with deployment
  • 13:30 slyngshede@dns1004: START - running authdns-update
  • 13:29 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti4006.ulsfo.wmnet
  • 13:28 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host tcp-proxy2002.codfw.wmnet
  • 13:28 arnaudb@cumin1004: START - Cookbook sre.gitlab.reboot-runner rolling reboot on P{gitlab-runner1003.eqiad.wmnet} and (A:gitlab-runner)
  • 13:27 seanleong-wmde@deploy1003: seanleong-wmde: Backport for Config change for opting out cswikiversity from LL suppression. (T441437) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 13:27 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host tcp-proxy2001.codfw.wmnet
  • 13:27 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-conf1004.eqiad.wmnet
  • 13:25 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host stewards1001.eqiad.wmnet
  • 13:25 arnaudb@cumin1004: END (PASS) - Cookbook sre.gitlab.reboot-runner (exit_code=0) rolling reboot on P{gitlab-runner1002.eqiad.wmnet} and (A:gitlab-runner)
  • 13:25 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host people2004.codfw.wmnet
  • 13:25 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host planet1003.eqiad.wmnet
  • 13:24 XioNoX: upgrade spicerack on cumin1004
  • 13:24 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti4007.ulsfo.wmnet
  • 13:24 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti4007.ulsfo.wmnet
  • 13:23 slyngshede@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host idp2005.wikimedia.org
  • 13:22 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host tcp-proxy2001.codfw.wmnet
  • 13:22 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host codesearch1001.eqiad.wmnet
  • 13:22 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host codesearch1001.eqiad.wmnet
  • 13:21 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host stewards1001.eqiad.wmnet
  • 13:21 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host an-conf1004.eqiad.wmnet
  • 13:21 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host people2004.codfw.wmnet
  • 13:21 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host planet1003.eqiad.wmnet
  • 13:20 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcumin1001.eqiad.wmnet
  • 13:20 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host tcp-proxy1002.eqiad.wmnet
  • 13:20 btullis@cumin1004: START - Cookbook sre.presto.reboot-workers for Presto an-presto cluster: Reboot Presto nodes
  • 13:19 slyngshede@cumin1004: START - Cookbook sre.hosts.reboot-single for host idp2005.wikimedia.org
  • 13:19 slyngshede@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host idp-test2005.wikimedia.org
  • 13:18 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1247', diff saved to https://phabricator.wikimedia.org/P96806 and previous config saved to /var/cache/conftool/dbconfig/20261019-131846-marostegui.json
  • 13:18 arnaudb@cumin1004: START - Cookbook sre.gitlab.reboot-runner rolling reboot on P{gitlab-runner1002.eqiad.wmnet} and (A:gitlab-runner)
  • 13:18 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcumin2001.codfw.wmnet
  • 13:17 tappof@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host prometheus1006.eqiad.wmnet
  • 13:17 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host zuul2003.codfw.wmnet
  • 13:16 filippo@cumin1004: START - Cookbook sre.hosts.reboot-single for host cloudcumin1001.eqiad.wmnet
  • 13:16 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host tcp-proxy1002.eqiad.wmnet
  • 13:16 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti7001.magru.wmnet
  • 13:16 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti4007.ulsfo.wmnet
  • 13:15 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host tcp-proxy1001.eqiad.wmnet
  • 13:15 slyngshede@cumin1004: START - Cookbook sre.hosts.reboot-single for host idp-test2005.wikimedia.org
  • 13:14 filippo@cumin1004: START - Cookbook sre.hosts.reboot-single for host cloudcumin2001.codfw.wmnet
  • 13:13 tappof@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host prometheus2006.codfw.wmnet
  • 13:13 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host zuul2003.codfw.wmnet
  • 13:12 slyngshede@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host idp-test1005.wikimedia.org
  • 13:12 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host zuul2002.codfw.wmnet
  • 13:11 tappof@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host prometheus2008.codfw.wmnet
  • 13:11 tappof@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host prometheus1008.eqiad.wmnet
  • 13:11 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host tcp-proxy1001.eqiad.wmnet
  • 13:08 slyngshede@cumin1004: START - Cookbook sre.hosts.reboot-single for host idp-test1005.wikimedia.org
  • 13:08 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host zuul2002.codfw.wmnet
  • 13:08 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudbackup2004.codfw.wmnet
  • 13:08 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host zuul2001.codfw.wmnet
  • 13:06 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti4007.ulsfo.wmnet
  • 13:06 seanleong-wmde@deploy1003: Started scap sync-world: Backport for Config change for opting out cswikiversity from LL suppression. (T441437)
  • 13:06 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti7001.magru.wmnet
  • 13:06 slyngshede@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host idm1001.wikimedia.org
  • 13:04 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host zuul2001.codfw.wmnet
  • 13:04 XioNoX: upgrade spicerack on cumin2003
  • 13:04 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host phab2003.codfw.wmnet
  • 13:04 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host contint2003.wikimedia.org
  • 13:03 tappof@cumin1004: START - Cookbook sre.hosts.reboot-single for host prometheus2008.codfw.wmnet
  • 13:03 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sretest1005.eqiad.wmnet
  • 13:03 tappof@cumin1004: START - Cookbook sre.hosts.reboot-single for host prometheus2006.codfw.wmnet
  • 13:03 tappof@cumin1004: START - Cookbook sre.hosts.reboot-single for host prometheus1008.eqiad.wmnet
  • 13:03 tappof@cumin1004: START - Cookbook sre.hosts.reboot-single for host prometheus1006.eqiad.wmnet
  • 13:03 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1247', diff saved to https://phabricator.wikimedia.org/P96805 and previous config saved to /var/cache/conftool/dbconfig/20261019-130344-marostegui.json
  • 13:03 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host releases2003.codfw.wmnet
  • 13:02 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host vrts2002.codfw.wmnet
  • 13:02 slyngshede@cumin1004: START - Cookbook sre.hosts.reboot-single for host idm1001.wikimedia.org
  • 13:01 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host people1005.eqiad.wmnet
  • 13:00 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti4008.ulsfo.wmnet
  • 13:00 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti4008.ulsfo.wmnet
  • 12:59 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host releases2003.codfw.wmnet
  • 12:59 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host lists2001.wikimedia.org
  • 12:59 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcephosd2007-dev.codfw.wmnet
  • 12:58 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host phab2003.codfw.wmnet
  • 12:57 filippo@cumin1004: START - Cookbook sre.hosts.reboot-single for host cloudbackup2004.codfw.wmnet
  • 12:57 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudbackup2003.codfw.wmnet
  • 12:57 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host people1005.eqiad.wmnet
  • 12:57 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host sretest1005.eqiad.wmnet
  • 12:57 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gitlab2003.wikimedia.org
  • 12:57 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host codesearch2001.codfw.wmnet
  • 12:57 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host contint2003.wikimedia.org
  • 12:57 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host phab1006.eqiad.wmnet
  • 12:56 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host stewards2001.codfw.wmnet
  • 12:56 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host codesearch2001.codfw.wmnet
  • 12:56 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host planet2003.codfw.wmnet
  • 12:56 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aphlict2001.codfw.wmnet
  • 12:56 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host vrts2002.codfw.wmnet
  • 12:55 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netflow2004.codfw.wmnet
  • 12:55 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host vrts1004.eqiad.wmnet
  • 12:55 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netflow2003.codfw.wmnet
  • 12:54 slyngshede@dns1004: END - running authdns-update
  • 12:54 moritzm: failover Ganeti master in magru to ganeti7004
  • 12:54 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti4008.ulsfo.wmnet
  • 12:53 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti7002.magru.wmnet
  • 12:53 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti7002.magru.wmnet
  • 12:52 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host stewards2001.codfw.wmnet
  • 12:52 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host planet2003.codfw.wmnet
  • 12:52 slyngshede@dns1004: START - running authdns-update
  • 12:52 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host lists2001.wikimedia.org
  • 12:52 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host aphlict2001.codfw.wmnet
  • 12:51 tappof@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host prometheus1005.eqiad.wmnet
  • 12:51 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host phab1006.eqiad.wmnet
  • 12:51 filippo@cumin1004: START - Cookbook sre.hosts.reboot-single for host cloudcephosd2007-dev.codfw.wmnet
  • 12:51 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host vrts1004.eqiad.wmnet
  • 12:51 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcephosd2006-dev.codfw.wmnet
  • 12:51 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host gitlab2003.wikimedia.org
  • 12:51 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netflow2003.codfw.wmnet
  • 12:50 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host doc2003.codfw.wmnet
  • 12:48 tappof@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host prometheus2005.codfw.wmnet
  • 12:48 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1247 (T441197)', diff saved to https://phabricator.wikimedia.org/P96804 and previous config saved to /var/cache/conftool/dbconfig/20261019-124842-marostegui.json
  • 12:48 slyngshede@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host idm2001.wikimedia.org
  • 12:47 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gerrit2002.wikimedia.org
  • 12:47 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti4008.ulsfo.wmnet
  • 12:47 tappof@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host prometheus1007.eqiad.wmnet
  • 12:47 tappof@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host prometheus2007.codfw.wmnet
  • 12:46 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netflow3004.esams.wmnet
  • 12:46 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host doc2003.codfw.wmnet
  • 12:46 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host doc1004.eqiad.wmnet
  • 12:45 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti7002.magru.wmnet
  • 12:44 slyngshede@cumin1004: START - Cookbook sre.hosts.reboot-single for host idm2001.wikimedia.org
  • 12:44 filippo@cumin1004: START - Cookbook sre.hosts.reboot-single for host cloudcephosd2006-dev.codfw.wmnet
  • 12:44 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcephosd2005-dev.codfw.wmnet
  • 12:43 filippo@cumin1004: START - Cookbook sre.hosts.reboot-single for host cloudbackup2003.codfw.wmnet
  • 12:43 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
  • 12:43 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudbackup1004.eqiad.wmnet
  • 12:42 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti7002.magru.wmnet
  • 12:42 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netflow3004.esams.wmnet
  • 12:42 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host doc1004.eqiad.wmnet
  • 12:41 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host gerrit2002.wikimedia.org
  • 12:41 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host etherpad2002.codfw.wmnet
  • 12:40 marostegui@cumin1004: dbctl commit (dc=all): 'Depooling db1247 (T441197)', diff saved to https://phabricator.wikimedia.org/P96803 and previous config saved to /var/cache/conftool/dbconfig/20261019-124039-marostegui.json
  • 12:40 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1247.eqiad.wmnet with reason: Maintenance
  • 12:40 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti7003.magru.wmnet
  • 12:40 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1244 (T441197)', diff saved to https://phabricator.wikimedia.org/P96802 and previous config saved to /var/cache/conftool/dbconfig/20261019-124022-marostegui.json
  • 12:40 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti7003.magru.wmnet
  • 12:39 slyngshede@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host idm-test1001.wikimedia.org
  • 12:39 tappof@cumin1004: START - Cookbook sre.hosts.reboot-single for host prometheus2007.codfw.wmnet
  • 12:39 tappof@cumin1004: START - Cookbook sre.hosts.reboot-single for host prometheus2005.codfw.wmnet
  • 12:39 tappof@cumin1004: START - Cookbook sre.hosts.reboot-single for host prometheus1007.eqiad.wmnet
  • 12:38 tappof@cumin1004: START - Cookbook sre.hosts.reboot-single for host prometheus1005.eqiad.wmnet
  • 12:37 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host etherpad2002.codfw.wmnet
  • 12:37 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host etherpad1004.eqiad.wmnet
  • 12:36 filippo@cumin1004: START - Cookbook sre.hosts.reboot-single for host cloudcephosd2005-dev.codfw.wmnet
  • 12:36 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netflow4003.ulsfo.wmnet
  • 12:36 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcephosd2004-dev.codfw.wmnet
  • 12:36 slyngshede@cumin1004: START - Cookbook sre.hosts.reboot-single for host idm-test1001.wikimedia.org
  • 12:35 tappof@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host prometheus7002.magru.wmnet
  • 12:35 ayounsi@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
  • 12:34 ayounsi@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
  • 12:34 ayounsi@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary
  • 12:34 filippo@cumin1004: START - Cookbook sre.hosts.reboot-single for host cloudbackup1004.eqiad.wmnet
  • 12:34 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudbackup1003.eqiad.wmnet
  • 12:33 slyngshede@cumin1004: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host idm-test1001.wikimedia.org
  • 12:33 slyngshede@cumin1004: START - Cookbook sre.hosts.reboot-single for host idm-test1001.wikimedia.org
  • 12:33 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host etherpad1004.eqiad.wmnet
  • 12:33 ayounsi@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
  • 12:33 arnaudb@cumin1004: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host etherpad1004.eqiad.wmnet
  • 12:32 ayounsi@cumin1004: END (ERROR) - Cookbook sre.netbox.update-extras (exit_code=97) rolling restart_daemons on A:netbox-canary
  • 12:32 ayounsi@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
  • 12:32 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host etherpad1004.eqiad.wmnet
  • 12:32 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netflow4003.ulsfo.wmnet
  • 12:31 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti7003.magru.wmnet
  • 12:30 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netflow5003.eqsin.wmnet
  • 12:29 tappof@cumin1004: START - Cookbook sre.hosts.reboot-single for host prometheus7002.magru.wmnet
  • 12:28 filippo@cumin1004: START - Cookbook sre.hosts.reboot-single for host cloudcephosd2004-dev.codfw.wmnet
  • 12:28 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcephmon2007-dev.codfw.wmnet
  • 12:28 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti7003.magru.wmnet
  • 12:26 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti7004.magru.wmnet
  • 12:26 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti7004.magru.wmnet
  • 12:26 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netflow5003.eqsin.wmnet
  • 12:26 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/analytics-test: apply
  • 12:25 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/analytics-test: apply
  • 12:25 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1244', diff saved to https://phabricator.wikimedia.org/P96801 and previous config saved to /var/cache/conftool/dbconfig/20261019-122520-marostegui.json
  • 12:25 filippo@cumin1004: START - Cookbook sre.hosts.reboot-single for host cloudbackup1003.eqiad.wmnet
  • 12:22 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netflow6001.drmrs.wmnet
  • 12:22 tappof@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host prometheus6002.drmrs.wmnet
  • 12:22 filippo@cumin1004: START - Cookbook sre.hosts.reboot-single for host cloudcephmon2007-dev.codfw.wmnet
  • 12:22 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcephmon2006-dev.codfw.wmnet
  • 12:20 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
  • 12:19 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
  • 12:18 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netflow6001.drmrs.wmnet
  • 12:17 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti7004.magru.wmnet
  • 12:16 tappof@cumin1004: START - Cookbook sre.hosts.reboot-single for host prometheus6002.drmrs.wmnet
  • 12:16 filippo@cumin1004: START - Cookbook sre.hosts.reboot-single for host cloudcephmon2006-dev.codfw.wmnet
  • 12:16 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcephmon2005-dev.codfw.wmnet
  • 12:15 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host failoid1003.eqiad.wmnet
  • 12:15 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netflow7002.magru.wmnet
  • 12:14 tappof@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host prometheus5003.eqsin.wmnet
  • 12:13 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudbackup1001-dev.eqiad.wmnet
  • 12:12 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
  • 12:12 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host failoid1003.eqiad.wmnet
  • 12:11 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply
  • 12:11 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host failoid2003.codfw.wmnet
  • 12:11 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netflow7002.magru.wmnet
  • 12:10 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti7004.magru.wmnet
  • 12:10 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1244', diff saved to https://phabricator.wikimedia.org/P96800 and previous config saved to /var/cache/conftool/dbconfig/20261019-121018-marostegui.json
  • 12:10 filippo@cumin1004: START - Cookbook sre.hosts.reboot-single for host cloudcephmon2005-dev.codfw.wmnet
  • 12:09 filippo@cumin1004: START - Cookbook sre.hosts.reboot-single for host cloudbackup1001-dev.eqiad.wmnet
  • 12:09 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudbackup1002-dev.eqiad.wmnet
  • 12:08 tappof@cumin1004: START - Cookbook sre.hosts.reboot-single for host prometheus5003.eqsin.wmnet
  • 12:07 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host failoid2003.codfw.wmnet
  • 12:07 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
  • 12:07 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply
  • 12:06 filippo@cumin1004: START - Cookbook sre.hosts.reboot-single for host cloudbackup1002-dev.eqiad.wmnet
  • 12:05 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcontrol2005-dev.codfw.wmnet
  • 12:00 tappof@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host prometheus4003.ulsfo.wmnet
  • 12:00 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
  • 12:00 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
  • 12:00 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
  • 12:00 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: sync
  • 11:59 filippo@cumin1004: START - Cookbook sre.hosts.reboot-single for host cloudcontrol2005-dev.codfw.wmnet
  • 11:59 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcontrol2006-dev.codfw.wmnet
  • 11:55 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1244 (T441197)', diff saved to https://phabricator.wikimedia.org/P96799 and previous config saved to /var/cache/conftool/dbconfig/20261019-115516-marostegui.json
  • 11:54 tappof@cumin1004: START - Cookbook sre.hosts.reboot-single for host prometheus4003.ulsfo.wmnet
  • 11:53 tappof@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host prometheus3004.esams.wmnet
  • 11:52 filippo@cumin1004: START - Cookbook sre.hosts.reboot-single for host cloudcontrol2006-dev.codfw.wmnet
  • 11:52 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcontrol2010-dev.codfw.wmnet
  • 11:47 tappof@cumin1004: START - Cookbook sre.hosts.reboot-single for host prometheus3004.esams.wmnet
  • 11:47 marostegui@cumin1004: dbctl commit (dc=all): 'Depooling db1244 (T441197)', diff saved to https://phabricator.wikimedia.org/P96798 and previous config saved to /var/cache/conftool/dbconfig/20261019-114728-marostegui.json
  • 11:47 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1244.eqiad.wmnet with reason: Maintenance
  • 11:47 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1242 (T441197)', diff saved to https://phabricator.wikimedia.org/P96797 and previous config saved to /var/cache/conftool/dbconfig/20261019-114711-marostegui.json
  • 11:45 filippo@cumin1004: START - Cookbook sre.hosts.reboot-single for host cloudcontrol2010-dev.codfw.wmnet
  • 11:45 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudgw2003-dev.codfw.wmnet
  • 11:40 filippo@cumin1004: START - Cookbook sre.hosts.reboot-single for host cloudgw2003-dev.codfw.wmnet
  • 11:40 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudgw2004-dev.codfw.wmnet
  • 11:34 filippo@cumin1004: START - Cookbook sre.hosts.reboot-single for host cloudgw2004-dev.codfw.wmnet
  • 11:34 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudidp2001-dev.codfw.wmnet
  • 11:32 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1242', diff saved to https://phabricator.wikimedia.org/P96795 and previous config saved to /var/cache/conftool/dbconfig/20261019-113208-marostegui.json
  • 11:31 moritzm: installing python3.11 security updates
  • 11:30 filippo@cumin1004: START - Cookbook sre.hosts.reboot-single for host cloudidp2001-dev.codfw.wmnet
  • 11:30 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudlb2002-dev.codfw.wmnet
  • 11:21 filippo@cumin1004: START - Cookbook sre.hosts.reboot-single for host cloudlb2002-dev.codfw.wmnet
  • 11:21 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudlb2003-dev.codfw.wmnet
  • 11:17 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1242', diff saved to https://phabricator.wikimedia.org/P96794 and previous config saved to /var/cache/conftool/dbconfig/20261019-111706-marostegui.json
  • 11:16 moritzm: installing xz-utils security updates
  • 11:12 filippo@cumin1004: START - Cookbook sre.hosts.reboot-single for host cloudlb2003-dev.codfw.wmnet
  • 11:12 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudlb2004-dev.codfw.wmnet
  • 11:09 moritzm: installing libevent security updates
  • 11:08 tappof@cumin1004: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling reboot on A:kafka-logging-eqiad
  • 11:03 filippo@cumin1004: START - Cookbook sre.hosts.reboot-single for host cloudlb2004-dev.codfw.wmnet
  • 11:03 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudnet2005-dev.codfw.wmnet
  • 11:02 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1242 (T441197)', diff saved to https://phabricator.wikimedia.org/P96793 and previous config saved to /var/cache/conftool/dbconfig/20261019-110204-marostegui.json
  • 10:55 filippo@cumin1004: START - Cookbook sre.hosts.reboot-single for host cloudnet2005-dev.codfw.wmnet
  • 10:55 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudnet2006-dev.codfw.wmnet
  • 10:53 marostegui@cumin1004: dbctl commit (dc=all): 'Depooling db1242 (T441197)', diff saved to https://phabricator.wikimedia.org/P96792 and previous config saved to /var/cache/conftool/dbconfig/20261019-105339-marostegui.json
  • 10:53 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1242.eqiad.wmnet with reason: Maintenance
  • 10:53 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1238 (T441197)', diff saved to https://phabricator.wikimedia.org/P96791 and previous config saved to /var/cache/conftool/dbconfig/20261019-105322-marostegui.json
  • 10:51 moritzm: installing openssl security updates
  • 10:48 filippo@cumin1004: START - Cookbook sre.hosts.reboot-single for host cloudnet2006-dev.codfw.wmnet
  • 10:48 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudrabbit2001-dev.codfw.wmnet
  • 10:41 filippo@cumin1004: START - Cookbook sre.hosts.reboot-single for host cloudrabbit2001-dev.codfw.wmnet
  • 10:41 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudrabbit2002-dev.codfw.wmnet
  • 10:39 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'llmisvc' .
  • 10:39 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
  • 10:38 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1238', diff saved to https://phabricator.wikimedia.org/P96790 and previous config saved to /var/cache/conftool/dbconfig/20261019-103820-marostegui.json
  • 10:37 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
  • 10:37 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
  • 10:32 filippo@cumin1004: START - Cookbook sre.hosts.reboot-single for host cloudrabbit2002-dev.codfw.wmnet
  • 10:32 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudrabbit2003-dev.codfw.wmnet
  • 10:27 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) upgrade A:liberica-magru
  • 10:26 filippo@cumin1004: START - Cookbook sre.hosts.reboot-single for host cloudrabbit2003-dev.codfw.wmnet
  • 10:25 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudservices2004-dev.codfw.wmnet
  • 10:25 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.upgrade upgrade A:liberica-magru
  • 10:23 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1238', diff saved to https://phabricator.wikimedia.org/P96789 and previous config saved to /var/cache/conftool/dbconfig/20261019-102318-marostegui.json
  • 10:21 vgutierrez: upload liberica 0.27 to bookworm-wikimedia (apt.wm.o)
  • 10:20 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
  • 10:19 filippo@cumin1004: START - Cookbook sre.hosts.reboot-single for host cloudservices2004-dev.codfw.wmnet
  • 10:19 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudservices2005-dev.codfw.wmnet
  • 10:18 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'.
  • 10:18 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'.
  • 10:17 tappof@cumin1004: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling reboot on A:kafka-logging-eqiad
  • 10:17 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'.
  • 10:16 tappof@cumin1004: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling reboot on A:kafka-logging-codfw
  • 10:16 cgoubert@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'.
  • 10:15 cgoubert@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'.
  • 10:14 cgoubert@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
  • 10:14 filippo@cumin1004: START - Cookbook sre.hosts.reboot-single for host cloudservices2005-dev.codfw.wmnet
  • 10:13 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudweb2002-dev.wikimedia.org
  • 10:13 cgoubert@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
  • 10:08 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1238 (T441197)', diff saved to https://phabricator.wikimedia.org/P96788 and previous config saved to /var/cache/conftool/dbconfig/20261019-100816-marostegui.json
  • 10:07 filippo@cumin1004: START - Cookbook sre.hosts.reboot-single for host cloudweb2002-dev.wikimedia.org
  • 10:00 marostegui@cumin1004: dbctl commit (dc=all): 'Depooling db1238 (T441197)', diff saved to https://phabricator.wikimedia.org/P96787 and previous config saved to /var/cache/conftool/dbconfig/20261019-100021-marostegui.json
  • 10:00 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1238.eqiad.wmnet with reason: Maintenance
  • 10:00 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1228 (T441197)', diff saved to https://phabricator.wikimedia.org/P96786 and previous config saved to /var/cache/conftool/dbconfig/20261019-100004-marostegui.json
  • 09:52 moritzm: installing Linux 6.1.190 on Bookworm hosts
  • 09:45 elukey: `sudo systemctl restart 'burrow-main-*'` on kafkamon2003
  • 09:45 blake@deploy1003: Failed scap sync-world: Enable k8s native sidecars (T417800) (duration: 12m 42s)
  • 09:45 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1228', diff saved to https://phabricator.wikimedia.org/P96784 and previous config saved to /var/cache/conftool/dbconfig/20261019-094502-marostegui.json
  • 09:43 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti-test2001.codfw.wmnet
  • 09:42 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'llmisvc' .
  • 09:42 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
  • 09:41 blake@deploy1003: Rolling back deployment
  • 09:37 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti-test2001.codfw.wmnet
  • 09:37 elukey: `sudo systemctl restart 'burrow-main-*'` on kafkamon1003 to see if the kafka mirror lag issues are burrow-related
  • 09:34 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
  • 09:33 blake@deploy1003: Started scap sync-world: Enable k8s native sidecars (T417800)
  • 09:30 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1228', diff saved to https://phabricator.wikimedia.org/P96783 and previous config saved to /var/cache/conftool/dbconfig/20261019-093000-marostegui.json
  • 09:29 tappof@cumin1004: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling reboot on A:kafka-logging-codfw
  • 09:15 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1228 (T441197)', diff saved to https://phabricator.wikimedia.org/P96782 and previous config saved to /var/cache/conftool/dbconfig/20261019-091458-marostegui.json
  • 09:07 marostegui@cumin1004: dbctl commit (dc=all): 'Depooling db1228 (T441197)', diff saved to https://phabricator.wikimedia.org/P96781 and previous config saved to /var/cache/conftool/dbconfig/20261019-090704-marostegui.json
  • 09:07 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1228.eqiad.wmnet with reason: Maintenance
  • 09:06 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1221 (T441197)', diff saved to https://phabricator.wikimedia.org/P96780 and previous config saved to /var/cache/conftool/dbconfig/20261019-090647-marostegui.json
  • 08:51 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1221', diff saved to https://phabricator.wikimedia.org/P96779 and previous config saved to /var/cache/conftool/dbconfig/20261019-085145-marostegui.json
  • 08:49 ayounsi@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
  • 08:48 ayounsi@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
  • 08:36 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1221', diff saved to https://phabricator.wikimedia.org/P96778 and previous config saved to /var/cache/conftool/dbconfig/20261019-083643-marostegui.json
  • 08:27 dkertesz@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on P{cp7011.*} and A:cp - upgrade haproxy to 3.2.26-1 (T441435)
  • 08:21 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1221 (T441197)', diff saved to https://phabricator.wikimedia.org/P96777 and previous config saved to /var/cache/conftool/dbconfig/20261019-082141-marostegui.json
  • 08:21 dkertesz@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on P{cp7011.*} and A:cp - upgrade haproxy to 3.2.26-1 (T441435)
  • 08:20 moritzm: installing Linux 6.12.113 on Trixie hosts
  • 08:19 dkertesz@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on P{cp7001.*} and A:cp - upgrade haproxy to 3.2.26-1 (T441435)
  • 08:14 dkertesz@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on P{cp7001.*} and A:cp - upgrade haproxy to 3.2.26-1 (T441435)
  • 08:08 marostegui@cumin1004: dbctl commit (dc=all): 'Remove db1156 T441298', diff saved to https://phabricator.wikimedia.org/P96776 and previous config saved to /var/cache/conftool/dbconfig/20261019-080831-marostegui.json
  • 08:07 dkertesz: fetching new haproxy packages on reprepro (3.2.25-1~bpo13+1 -> 3.2.26-1~bpo13+1) - T441435
  • 08:00 ayounsi@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary
  • 07:59 ayounsi@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
  • 07:57 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'llmisvc' .
  • 07:57 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
  • 07:42 Emperor: cephadm shell -- radosgw-admin quota set --quota-scope=user --uid=gitlab-rw --max-size=400G [apus]
  • 07:21 marostegui@cumin1004: dbctl commit (dc=all): 'Depooling db1221 (T441197)', diff saved to https://phabricator.wikimedia.org/P96775 and previous config saved to /var/cache/conftool/dbconfig/20261019-072128-marostegui.json
  • 07:21 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on 6 hosts with reason: Maintenance
  • 07:21 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1221.eqiad.wmnet with reason: Maintenance
  • 07:21 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1199 (T441197)', diff saved to https://phabricator.wikimedia.org/P96774 and previous config saved to /var/cache/conftool/dbconfig/20261019-072059-marostegui.json
  • 07:05 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1199', diff saved to https://phabricator.wikimedia.org/P96773 and previous config saved to /var/cache/conftool/dbconfig/20261019-070557-marostegui.json
  • 06:50 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1199', diff saved to https://phabricator.wikimedia.org/P96772 and previous config saved to /var/cache/conftool/dbconfig/20261019-065055-marostegui.json
  • 06:35 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1199 (T441197)', diff saved to https://phabricator.wikimedia.org/P96771 and previous config saved to /var/cache/conftool/dbconfig/20261019-063553-marostegui.json
  • 06:26 marostegui@cumin1004: dbctl commit (dc=all): 'Depooling db1199 (T441197)', diff saved to https://phabricator.wikimedia.org/P96770 and previous config saved to /var/cache/conftool/dbconfig/20261019-062627-marostegui.json
  • 06:26 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1199.eqiad.wmnet with reason: Maintenance
  • 06:19 kart_: machinetranslation: Update to 2026-09-21-112314-production (T437213)
  • 06:17 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2179.codfw.wmnet with reason: Maintenance
  • 05:48 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on clouddb1023.eqiad.wmnet with reason: Testing
  • 05:34 kartik@deploy1003: helmfile [eqiad] DONE helmfile.d/services/machinetranslation: apply
  • 05:29 kartik@deploy1003: helmfile [eqiad] START helmfile.d/services/machinetranslation: apply
  • 05:26 kartik@deploy1003: helmfile [codfw] DONE helmfile.d/services/machinetranslation: apply
  • 05:22 kartik@deploy1003: helmfile [codfw] START helmfile.d/services/machinetranslation: apply
  • 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image
  • 00:00 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
  • 00:00 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: sync

2026-10-18

  • 12:00 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
  • 12:00 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
  • 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 29s)
  • 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image
  • 00:00 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
  • 00:00 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
  • 00:00 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
  • 00:00 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: sync

2026-10-17

  • 12:00 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
  • 12:00 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
  • 12:00 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
  • 12:00 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: sync
  • 04:26 vgutierrez@cumin1004: END (FAIL) - Cookbook sre.mysql.depool (exit_code=99) depool db2249: Depooling replica
  • 04:26 vgutierrez@cumin1004: START - Cookbook sre.mysql.depool depool db2249: Depooling replica
  • 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 26s)
  • 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image
  • 00:00 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
  • 00:00 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: sync

2026-10-16

  • 16:43 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2240 (T441197)', diff saved to https://phabricator.wikimedia.org/P96764 and previous config saved to /var/cache/conftool/dbconfig/20261016-164339-marostegui.json
  • 16:28 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2240', diff saved to https://phabricator.wikimedia.org/P96763 and previous config saved to /var/cache/conftool/dbconfig/20261016-162837-marostegui.json
  • 16:13 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2240', diff saved to https://phabricator.wikimedia.org/P96762 and previous config saved to /var/cache/conftool/dbconfig/20261016-161335-marostegui.json
  • 16:02 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp2047.*
  • 15:58 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2240 (T441197)', diff saved to https://phabricator.wikimedia.org/P96761 and previous config saved to /var/cache/conftool/dbconfig/20261016-155833-marostegui.json
  • 15:50 marostegui@cumin1004: dbctl commit (dc=all): 'Depooling db2240 (T441197)', diff saved to https://phabricator.wikimedia.org/P96760 and previous config saved to /var/cache/conftool/dbconfig/20261016-155040-marostegui.json
  • 15:50 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2240.codfw.wmnet with reason: Maintenance
  • 15:42 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2239.codfw.wmnet with reason: Maintenance
  • 15:41 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2237 (T441197)', diff saved to https://phabricator.wikimedia.org/P96759 and previous config saved to /var/cache/conftool/dbconfig/20261016-154144-marostegui.json
  • 15:31 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2203.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 15:30 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2203.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 15:26 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2237', diff saved to https://phabricator.wikimedia.org/P96758 and previous config saved to /var/cache/conftool/dbconfig/20261016-152642-marostegui.json
  • 15:11 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2237', diff saved to https://phabricator.wikimedia.org/P96757 and previous config saved to /var/cache/conftool/dbconfig/20261016-151140-marostegui.json
  • 14:56 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2237 (T441197)', diff saved to https://phabricator.wikimedia.org/P96755 and previous config saved to /var/cache/conftool/dbconfig/20261016-145638-marostegui.json
  • 14:48 marostegui@cumin1004: dbctl commit (dc=all): 'Depooling db2237 (T441197)', diff saved to https://phabricator.wikimedia.org/P96754 and previous config saved to /var/cache/conftool/dbconfig/20261016-144844-marostegui.json
  • 14:48 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2237.codfw.wmnet with reason: Maintenance
  • 14:48 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2236 (T441197)', diff saved to https://phabricator.wikimedia.org/P96753 and previous config saved to /var/cache/conftool/dbconfig/20261016-144826-marostegui.json
  • 14:33 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2236', diff saved to https://phabricator.wikimedia.org/P96752 and previous config saved to /var/cache/conftool/dbconfig/20261016-143324-marostegui.json
  • 14:29 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1309.eqiad.wmnet
  • 14:29 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1309.eqiad.wmnet
  • 14:21 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
  • 14:21 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1309.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 14:19 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'.
  • 14:19 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'.
  • 14:19 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
  • 14:18 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2236', diff saved to https://phabricator.wikimedia.org/P96751 and previous config saved to /var/cache/conftool/dbconfig/20261016-141822-marostegui.json
  • 14:18 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
  • 14:10 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply
  • 14:10 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply
  • 14:09 cwilliams@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2158: Testing Zarcillo
  • 14:03 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1309.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 14:03 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1302.eqiad.wmnet
  • 14:03 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1302.eqiad.wmnet
  • 14:03 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2236 (T441197)', diff saved to https://phabricator.wikimedia.org/P96748 and previous config saved to /var/cache/conftool/dbconfig/20261016-140320-marostegui.json
  • 13:58 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/analytics-test: apply
  • 13:58 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/analytics-test: apply
  • 13:58 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wmde: apply
  • 13:58 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wmde: apply
  • 13:57 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wikidata: apply
  • 13:57 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wikidata: apply
  • 13:57 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
  • 13:56 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
  • 13:56 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-sre: apply
  • 13:56 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-sre: apply
  • 13:56 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-search: apply
  • 13:55 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-search: apply
  • 13:55 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-research: apply
  • 13:55 marostegui@cumin1004: dbctl commit (dc=all): 'Depooling db2236 (T441197)', diff saved to https://phabricator.wikimedia.org/P96747 and previous config saved to /var/cache/conftool/dbconfig/20261016-135522-marostegui.json
  • 13:55 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2236.codfw.wmnet with reason: Maintenance
  • 13:55 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-research: apply
  • 13:55 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-platform-eng: apply
  • 13:55 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2219 (T441197)', diff saved to https://phabricator.wikimedia.org/P96746 and previous config saved to /var/cache/conftool/dbconfig/20261016-135504-marostegui.json
  • 13:54 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-platform-eng: apply
  • 13:54 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
  • 13:54 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply
  • 13:54 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply
  • 13:53 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply
  • 13:53 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-fr-tech: apply
  • 13:53 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-fr-tech: apply
  • 13:53 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-experiment-platform: apply
  • 13:52 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-experiment-platform: apply
  • 13:52 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
  • 13:51 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
  • 13:51 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply
  • 13:51 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply
  • 13:51 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-product: apply
  • 13:50 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-product: apply
  • 13:50 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/blunderbuss: apply
  • 13:50 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1298.eqiad.wmnet with OS bookworm
  • 13:49 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/blunderbuss: apply
  • 13:49 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthbook: apply
  • 13:49 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook: apply
  • 13:49 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply
  • 13:49 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply
  • 13:49 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/superset: apply
  • 13:48 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/superset: apply
  • 13:48 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/superset-next: apply
  • 13:47 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/superset-next: apply
  • 13:47 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/spark-history: apply
  • 13:46 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/spark-history: apply
  • 13:46 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/spark-history: apply
  • 13:45 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/spark-history: apply
  • 13:41 Daimona: Running query from T441353#12441735 on x1.wikishared to restore accidentally deleted event participants
  • 13:40 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2219', diff saved to https://phabricator.wikimedia.org/P96744 and previous config saved to /var/cache/conftool/dbconfig/20261016-134002-marostegui.json
  • 13:32 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) restart P{lvs7003.magru.wmnet} and A:liberica
  • 13:32 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) pooling P{lvs7003.magru.wmnet} and A:liberica
  • 13:32 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin pooling P{lvs7003.magru.wmnet} and A:liberica
  • 13:31 jclark@cumin1004: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on wikikube-worker1298.eqiad.wmnet with reason: host reimage
  • 13:31 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) depooling P{lvs7003.magru.wmnet} and A:liberica
  • 13:31 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin depooling P{lvs7003.magru.wmnet} and A:liberica
  • 13:31 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.upgrade restart P{lvs7003.magru.wmnet} and A:liberica
  • 13:31 jclark@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1298.eqiad.wmnet with reason: host reimage
  • 13:25 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1302.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 13:25 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2219', diff saved to https://phabricator.wikimedia.org/P96741 and previous config saved to /var/cache/conftool/dbconfig/20261016-132500-marostegui.json
  • 13:23 cwilliams@cumin1004: START - Cookbook sre.mysql.pool pool db2158: Testing Zarcillo
  • 13:22 cwilliams@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2158: Testing Zarcillo
  • 13:21 cwilliams@cumin1004: START - Cookbook sre.mysql.depool depool db2158: Testing Zarcillo
  • 13:15 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) restart P{lvs7003.magru.wmnet} and A:liberica
  • 13:15 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) pooling P{lvs7003.magru.wmnet} and A:liberica
  • 13:15 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin pooling P{lvs7003.magru.wmnet} and A:liberica
  • 13:14 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) depooling P{lvs7003.magru.wmnet} and A:liberica
  • 13:14 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin depooling P{lvs7003.magru.wmnet} and A:liberica
  • 13:14 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.upgrade restart P{lvs7003.magru.wmnet} and A:liberica
  • 13:11 jclark@cumin1004: START - Cookbook sre.hosts.reimage for host wikikube-worker1298.eqiad.wmnet with OS bookworm
  • 13:11 claime: Raising extstore item_size to 280 for codfw memcached - T440896
  • 13:10 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2219 (T441197)', diff saved to https://phabricator.wikimedia.org/P96738 and previous config saved to /var/cache/conftool/dbconfig/20261016-130958-marostegui.json
  • 13:05 cwilliams@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
  • 13:04 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1302.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 13:01 marostegui@cumin1004: dbctl commit (dc=all): 'Depooling db2219 (T441197)', diff saved to https://phabricator.wikimedia.org/P96737 and previous config saved to /var/cache/conftool/dbconfig/20261016-130151-marostegui.json
  • 13:01 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2219.codfw.wmnet with reason: Maintenance
  • 13:01 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2210 (T441197)', diff saved to https://phabricator.wikimedia.org/P96736 and previous config saved to /var/cache/conftool/dbconfig/20261016-130133-marostegui.json
  • 12:46 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2210', diff saved to https://phabricator.wikimedia.org/P96735 and previous config saved to /var/cache/conftool/dbconfig/20261016-124631-marostegui.json
  • 12:31 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2210', diff saved to https://phabricator.wikimedia.org/P96734 and previous config saved to /var/cache/conftool/dbconfig/20261016-123130-marostegui.json
  • 12:23 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-sre: apply
  • 12:23 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-sre: apply
  • 12:16 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2210 (T441197)', diff saved to https://phabricator.wikimedia.org/P96733 and previous config saved to /var/cache/conftool/dbconfig/20261016-121628-marostegui.json
  • 12:08 marostegui@cumin1004: dbctl commit (dc=all): 'Depooling db2210 (T441197)', diff saved to https://phabricator.wikimedia.org/P96732 and previous config saved to /var/cache/conftool/dbconfig/20261016-120839-marostegui.json
  • 12:08 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2210.codfw.wmnet with reason: Maintenance
  • 12:08 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2206 (T441197)', diff saved to https://phabricator.wikimedia.org/P96731 and previous config saved to /var/cache/conftool/dbconfig/20261016-120821-marostegui.json
  • 12:01 jclark@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker1298.eqiad.wmnet with OS bookworm
  • 12:00 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
  • 12:00 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
  • 12:00 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
  • 12:00 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
  • 11:53 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2206', diff saved to https://phabricator.wikimedia.org/P96730 and previous config saved to /var/cache/conftool/dbconfig/20261016-115319-marostegui.json
  • 11:38 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2206', diff saved to https://phabricator.wikimedia.org/P96729 and previous config saved to /var/cache/conftool/dbconfig/20261016-113817-marostegui.json
  • 11:24 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
  • 11:23 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2206 (T441197)', diff saved to https://phabricator.wikimedia.org/P96728 and previous config saved to /var/cache/conftool/dbconfig/20261016-112315-marostegui.json
  • 11:14 marostegui@cumin1004: dbctl commit (dc=all): 'Depooling db2206 (T441197)', diff saved to https://phabricator.wikimedia.org/P96727 and previous config saved to /var/cache/conftool/dbconfig/20261016-111455-marostegui.json
  • 11:14 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2206.codfw.wmnet with reason: Maintenance
  • 11:11 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.enable_virtualization (exit_code=0) rolling provision on D{wikikube-worker[1350-1379].eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad)
  • 11:11 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1379.eqiad.wmnet
  • 11:11 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1379.eqiad.wmnet
  • 11:10 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1379.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 11:06 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2199.codfw.wmnet with reason: Maintenance
  • 11:05 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2172 (T441197)', diff saved to https://phabricator.wikimedia.org/P96726 and previous config saved to /var/cache/conftool/dbconfig/20261016-110555-marostegui.json
  • 11:04 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1379.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 11:03 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1379.eqiad.wmnet
  • 11:03 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1379.eqiad.wmnet
  • 11:02 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1378.eqiad.wmnet
  • 11:02 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1378.eqiad.wmnet
  • 10:50 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2172', diff saved to https://phabricator.wikimedia.org/P96725 and previous config saved to /var/cache/conftool/dbconfig/20261016-105054-marostegui.json
  • 10:36 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.enable_virtualization (exit_code=0) rolling provision on D{wikikube-worker[2341-2364].codfw.wmnet} and (A:wikikube-master-codfw or A:wikikube-worker-codfw)
  • 10:36 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2364.codfw.wmnet
  • 10:36 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2364.codfw.wmnet
  • 10:36 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1378.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 10:35 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2172', diff saved to https://phabricator.wikimedia.org/P96724 and previous config saved to /var/cache/conftool/dbconfig/20261016-103551-marostegui.json
  • 10:34 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2364.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 10:34 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
  • 10:29 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1378.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 10:29 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1378.eqiad.wmnet
  • 10:28 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1378.eqiad.wmnet
  • 10:28 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1377.eqiad.wmnet
  • 10:28 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1377.eqiad.wmnet
  • 10:28 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2364.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 10:27 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2364.codfw.wmnet
  • 10:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2364.codfw.wmnet
  • 10:26 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2363.codfw.wmnet
  • 10:26 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2363.codfw.wmnet
  • 10:26 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2363.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 10:20 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2172 (T441197)', diff saved to https://phabricator.wikimedia.org/P96723 and previous config saved to /var/cache/conftool/dbconfig/20261016-102049-marostegui.json
  • 10:19 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2363.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 10:19 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2363.codfw.wmnet
  • 10:18 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2363.codfw.wmnet
  • 10:18 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2362.codfw.wmnet
  • 10:17 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1377.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 10:17 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2362.codfw.wmnet
  • 10:17 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2362.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 10:12 marostegui@cumin1004: dbctl commit (dc=all): 'Depooling db2172 (T441197)', diff saved to https://phabricator.wikimedia.org/P96722 and previous config saved to /var/cache/conftool/dbconfig/20261016-101209-marostegui.json
  • 10:12 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2172.codfw.wmnet with reason: Maintenance
  • 10:11 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2155 (T441197)', diff saved to https://phabricator.wikimedia.org/P96721 and previous config saved to /var/cache/conftool/dbconfig/20261016-101151-marostegui.json
  • 10:11 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1377.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 10:10 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2362.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 10:05 oblivian@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-worker1377.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 10:04 oblivian@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-worker2362.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 10:01 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1377.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 10:01 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2362.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 10:01 jayme@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for kafka-main2008.codfw.wmnet
  • 10:01 jayme@cumin1004: START - Cookbook sre.hosts.remove-downtime for kafka-main2008.codfw.wmnet
  • 10:00 oblivian@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-worker1377.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 09:57 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1377.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 09:56 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1377.eqiad.wmnet
  • 09:56 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2155', diff saved to https://phabricator.wikimedia.org/P96720 and previous config saved to /var/cache/conftool/dbconfig/20261016-095650-marostegui.json
  • 09:56 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1377.eqiad.wmnet
  • 09:56 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1376.eqiad.wmnet
  • 09:56 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1376.eqiad.wmnet
  • 09:54 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1376.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 09:53 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
  • 09:52 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
  • 09:47 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1376.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 09:47 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1376.eqiad.wmnet
  • 09:47 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1376.eqiad.wmnet
  • 09:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1375.eqiad.wmnet
  • 09:46 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1375.eqiad.wmnet
  • 09:41 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2155', diff saved to https://phabricator.wikimedia.org/P96719 and previous config saved to /var/cache/conftool/dbconfig/20261016-094147-marostegui.json
  • 09:38 jayme@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on kafka-main2008.codfw.wmnet with reason: firmware upgrade
  • 09:37 jayme@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for kafka-main1009.eqiad.wmnet
  • 09:37 jayme@cumin1004: START - Cookbook sre.hosts.remove-downtime for kafka-main1009.eqiad.wmnet
  • 09:35 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1375.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 09:29 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1375.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 09:28 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1375.eqiad.wmnet
  • 09:28 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1375.eqiad.wmnet
  • 09:28 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1374.eqiad.wmnet
  • 09:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1374.eqiad.wmnet
  • 09:27 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1374.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 09:26 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2155 (T441197)', diff saved to https://phabricator.wikimedia.org/P96718 and previous config saved to /var/cache/conftool/dbconfig/20261016-092645-marostegui.json
  • 09:20 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1374.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 09:20 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1374.eqiad.wmnet
  • 09:19 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1374.eqiad.wmnet
  • 09:19 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1373.eqiad.wmnet
  • 09:19 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1373.eqiad.wmnet
  • 09:18 marostegui@cumin1004: dbctl commit (dc=all): 'Depooling db2155 (T441197)', diff saved to https://phabricator.wikimedia.org/P96717 and previous config saved to /var/cache/conftool/dbconfig/20261016-091831-marostegui.json
  • 09:18 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2155.codfw.wmnet with reason: Maintenance
  • 09:17 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1184.eqiad.wmnet with reason: Maintenance
  • 09:13 jayme@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on kafka-main1009.eqiad.wmnet with reason: firmware upgrade
  • 09:12 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on dbstore1008.eqiad.wmnet with reason: Maintenance
  • 09:12 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1283.eqiad.wmnet with reason: Maintenance
  • 09:11 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1277.eqiad.wmnet with reason: Maintenance
  • 09:11 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1251.eqiad.wmnet with reason: Maintenance
  • 09:11 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1240.eqiad.wmnet with reason: Maintenance
  • 09:10 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1239.eqiad.wmnet with reason: Maintenance
  • 09:10 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1235.eqiad.wmnet with reason: Maintenance
  • 09:10 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1234.eqiad.wmnet with reason: Maintenance
  • 09:10 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1232.eqiad.wmnet with reason: Maintenance
  • 09:09 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1219.eqiad.wmnet with reason: Maintenance
  • 09:09 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1218.eqiad.wmnet with reason: Maintenance
  • 09:09 XioNoX: test setting 192.168.0.0/24 IP on mr1-eqiad:irb.900
  • 09:09 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1206.eqiad.wmnet with reason: Maintenance
  • 09:08 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 6 hosts with reason: Maintenance
  • 09:08 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1196.eqiad.wmnet with reason: Maintenance
  • 09:08 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1373.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 09:08 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1195.eqiad.wmnet with reason: Maintenance
  • 09:07 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1186.eqiad.wmnet with reason: Maintenance
  • 09:07 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1169.eqiad.wmnet with reason: Maintenance
  • 09:07 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1163.eqiad.wmnet with reason: Maintenance
  • 09:06 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2203.codfw.wmnet with reason: Maintenance
  • 09:05 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2250.codfw.wmnet with reason: Maintenance
  • 09:05 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2216.codfw.wmnet with reason: Maintenance
  • 09:05 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2212.codfw.wmnet with reason: Maintenance
  • 09:04 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2202.codfw.wmnet with reason: Maintenance
  • 09:04 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2188.codfw.wmnet with reason: Maintenance
  • 09:04 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2176.codfw.wmnet with reason: Maintenance
  • 09:03 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2174.codfw.wmnet with reason: Maintenance
  • 09:03 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2173.codfw.wmnet with reason: Maintenance
  • 09:03 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2170.codfw.wmnet with reason: Maintenance
  • 09:03 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2153.codfw.wmnet with reason: Maintenance
  • 09:01 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1210.eqiad.wmnet with reason: Maintenance
  • 09:01 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1373.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 09:01 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1373.eqiad.wmnet
  • 09:01 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on dbstore1008.eqiad.wmnet with reason: Maintenance
  • 09:00 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1373.eqiad.wmnet
  • 09:00 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 7 hosts with reason: Maintenance
  • 09:00 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1372.eqiad.wmnet
  • 09:00 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1372.eqiad.wmnet
  • 09:00 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1275.eqiad.wmnet with reason: Maintenance
  • 08:59 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1372.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 08:59 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1274.eqiad.wmnet with reason: Maintenance
  • 08:59 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1230.eqiad.wmnet with reason: Maintenance
  • 08:58 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1216.eqiad.wmnet with reason: Maintenance
  • 08:57 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1207.eqiad.wmnet with reason: Maintenance
  • 08:57 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1200.eqiad.wmnet with reason: Maintenance
  • 08:56 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1185.eqiad.wmnet with reason: Maintenance
  • 08:55 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1161.eqiad.wmnet with reason: Maintenance
  • 08:55 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1159.eqiad.wmnet with reason: Maintenance
  • 08:53 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1372.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 08:53 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1372.eqiad.wmnet
  • 08:52 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1372.eqiad.wmnet
  • 08:52 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2157.codfw.wmnet with reason: Maintenance
  • 08:52 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1371.eqiad.wmnet
  • 08:52 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1371.eqiad.wmnet
  • 08:48 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2223.codfw.wmnet with reason: Maintenance
  • 08:46 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2213.codfw.wmnet with reason: Maintenance
  • 08:44 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2211.codfw.wmnet with reason: Maintenance
  • 08:42 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2201.codfw.wmnet with reason: Maintenance
  • 08:41 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1371.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 08:40 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2192.codfw.wmnet with reason: Maintenance
  • 08:38 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2178.codfw.wmnet with reason: Maintenance
  • 08:37 oblivian@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-worker2362.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 08:35 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2171.codfw.wmnet with reason: Maintenance
  • 08:34 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1371.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 08:34 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1371.eqiad.wmnet
  • 08:33 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2362.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 08:33 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1371.eqiad.wmnet
  • 08:33 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2362.codfw.wmnet
  • 08:33 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1370.eqiad.wmnet
  • 08:33 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1370.eqiad.wmnet
  • 08:32 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2362.codfw.wmnet
  • 08:32 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2361.codfw.wmnet
  • 08:32 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2361.codfw.wmnet
  • 08:32 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2361.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 08:28 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db1189.eqiad.wmnet with reason: Maintenance
  • 08:27 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.decommission (exit_code=0)
  • 08:27 marostegui@cumin1004: Removing clouddb1013 from zarcillo T441300
  • 08:26 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts clouddb1013.eqiad.wmnet
  • 08:26 marostegui@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 08:26 marostegui@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: clouddb1013.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1004"
  • 08:26 marostegui@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: clouddb1013.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1004"
  • 08:25 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2361.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 08:25 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2361.codfw.wmnet
  • 08:24 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2361.codfw.wmnet
  • 08:24 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2360.codfw.wmnet
  • 08:24 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2360.codfw.wmnet
  • 08:22 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1370.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 08:22 marostegui@cumin1004: START - Cookbook sre.dns.netbox
  • 08:22 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
  • 08:19 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db1289.eqiad.wmnet with reason: Maintenance
  • 08:17 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db1280.eqiad.wmnet with reason: Maintenance
  • 08:16 marostegui@cumin1004: START - Cookbook sre.hosts.decommission for hosts clouddb1013.eqiad.wmnet
  • 08:16 marostegui@cumin1004: START - Cookbook sre.mysql.decommission
  • 08:15 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1370.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 08:15 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1370.eqiad.wmnet
  • 08:15 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db1272.eqiad.wmnet with reason: Maintenance
  • 08:15 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1370.eqiad.wmnet
  • 08:14 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1369.eqiad.wmnet
  • 08:14 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1369.eqiad.wmnet
  • 08:13 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2360.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 08:13 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1369.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 08:12 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db1265.eqiad.wmnet with reason: Maintenance
  • 08:09 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db1240.eqiad.wmnet with reason: Maintenance
  • 08:07 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db1223.eqiad.wmnet with reason: Maintenance
  • 08:06 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2360.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 08:06 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1369.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 08:06 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2360.codfw.wmnet
  • 08:06 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1369.eqiad.wmnet
  • 08:06 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2360.codfw.wmnet
  • 08:05 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1369.eqiad.wmnet
  • 08:05 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2359.codfw.wmnet
  • 08:05 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2359.codfw.wmnet
  • 08:05 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1368.eqiad.wmnet
  • 08:05 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1368.eqiad.wmnet
  • 08:04 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on 7 hosts with reason: Maintenance
  • 08:04 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1368.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 08:04 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db1212.eqiad.wmnet with reason: Maintenance
  • 08:01 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db1198.eqiad.wmnet with reason: Maintenance
  • 07:58 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db1175.eqiad.wmnet with reason: Maintenance
  • 07:57 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1368.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 07:57 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1368.eqiad.wmnet
  • 07:56 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1368.eqiad.wmnet
  • 07:56 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1367.eqiad.wmnet
  • 07:56 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1367.eqiad.wmnet
  • 07:55 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db1166.eqiad.wmnet with reason: Maintenance
  • 07:53 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db1157.eqiad.wmnet with reason: Maintenance
  • 07:41 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1367.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 07:35 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db2205.codfw.wmnet with reason: Maintenance
  • 07:34 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1367.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 07:34 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1367.eqiad.wmnet
  • 07:33 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1367.eqiad.wmnet
  • 07:33 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1366.eqiad.wmnet
  • 07:33 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1366.eqiad.wmnet
  • 07:32 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2359.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 07:25 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2359.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 07:25 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2359.codfw.wmnet
  • 07:24 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2359.codfw.wmnet
  • 07:24 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2358.codfw.wmnet
  • 07:24 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2358.codfw.wmnet
  • 07:24 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2358.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 07:22 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1366.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 07:17 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2358.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 07:17 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2358.codfw.wmnet
  • 07:16 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2358.codfw.wmnet
  • 07:16 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2357.codfw.wmnet
  • 07:16 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2357.codfw.wmnet
  • 07:15 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2357.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 07:15 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1366.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 07:15 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1366.eqiad.wmnet
  • 07:14 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1366.eqiad.wmnet
  • 07:14 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1365.eqiad.wmnet
  • 07:14 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1365.eqiad.wmnet
  • 07:12 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db2239.codfw.wmnet with reason: Maintenance
  • 07:12 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1365.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 07:09 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2357.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 07:08 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2357.codfw.wmnet
  • 07:08 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2357.codfw.wmnet
  • 07:08 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2356.codfw.wmnet
  • 07:07 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2356.codfw.wmnet
  • 07:07 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2356.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 07:06 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1365.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 07:05 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1365.eqiad.wmnet
  • 07:05 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1365.eqiad.wmnet
  • 07:05 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1364.eqiad.wmnet
  • 07:05 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1364.eqiad.wmnet
  • 07:03 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1364.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 07:00 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2356.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 07:00 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2356.codfw.wmnet
  • 06:59 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db2227.codfw.wmnet with reason: Maintenance
  • 06:59 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2356.codfw.wmnet
  • 06:58 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2355.codfw.wmnet
  • 06:58 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2355.codfw.wmnet
  • 06:57 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2355.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 06:57 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1364.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 06:57 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1364.eqiad.wmnet
  • 06:56 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1364.eqiad.wmnet
  • 06:56 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1363.eqiad.wmnet
  • 06:56 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1363.eqiad.wmnet
  • 06:51 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2355.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 06:50 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2355.codfw.wmnet
  • 06:50 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2355.codfw.wmnet
  • 06:50 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2354.codfw.wmnet
  • 06:50 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2354.codfw.wmnet
  • 06:49 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2354.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 06:46 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db2209.codfw.wmnet with reason: Maintenance
  • 06:43 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2354.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 06:42 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2354.codfw.wmnet
  • 06:42 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2354.codfw.wmnet
  • 06:42 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2353.codfw.wmnet
  • 06:41 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2353.codfw.wmnet
  • 06:40 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1363.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 06:40 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2353.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 06:34 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2353.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 06:34 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1363.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 06:33 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2353.codfw.wmnet
  • 06:33 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1363.eqiad.wmnet
  • 06:33 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db2194.codfw.wmnet with reason: Maintenance
  • 06:33 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2353.codfw.wmnet
  • 06:33 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1363.eqiad.wmnet
  • 06:32 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2352.codfw.wmnet
  • 06:32 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1362.eqiad.wmnet
  • 06:32 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1362.eqiad.wmnet
  • 06:32 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2352.codfw.wmnet
  • 06:32 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2352.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 06:31 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1362.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 06:25 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2352.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 06:25 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2352.codfw.wmnet
  • 06:25 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1362.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 06:24 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1362.eqiad.wmnet
  • 06:24 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2352.codfw.wmnet
  • 06:24 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2351.codfw.wmnet
  • 06:24 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2351.codfw.wmnet
  • 06:24 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1362.eqiad.wmnet
  • 06:23 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1361.eqiad.wmnet
  • 06:23 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1361.eqiad.wmnet
  • 06:23 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2351.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 06:23 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1361.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 06:20 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db2190.codfw.wmnet with reason: Maintenance
  • 06:17 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2351.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 06:16 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2351.codfw.wmnet
  • 06:16 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1361.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 06:16 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1361.eqiad.wmnet
  • 06:15 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1361.eqiad.wmnet
  • 06:15 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1360.eqiad.wmnet
  • 06:15 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2351.codfw.wmnet
  • 06:15 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1360.eqiad.wmnet
  • 06:15 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2350.codfw.wmnet
  • 06:15 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2350.codfw.wmnet
  • 06:14 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2350.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 06:08 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2350.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 06:08 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2350.codfw.wmnet
  • 06:07 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1360.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 06:07 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db2177.codfw.wmnet with reason: Maintenance
  • 06:07 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2350.codfw.wmnet
  • 06:07 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2349.codfw.wmnet
  • 06:07 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2349.codfw.wmnet
  • 06:06 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2349.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 06:01 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1360.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 06:00 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1360.eqiad.wmnet
  • 06:00 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2349.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 06:00 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1360.eqiad.wmnet
  • 05:59 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1359.eqiad.wmnet
  • 05:59 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1359.eqiad.wmnet
  • 05:59 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2349.codfw.wmnet
  • 05:59 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2349.codfw.wmnet
  • 05:59 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2348.codfw.wmnet
  • 05:59 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2348.codfw.wmnet
  • 05:57 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1359.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 05:57 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2348.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 05:54 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db2156.codfw.wmnet with reason: Maintenance
  • 05:53 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db2200.codfw.wmnet with reason: Maintenance
  • 05:53 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db2198.codfw.wmnet with reason: Maintenance
  • 05:53 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db2182.codfw.wmnet with reason: Maintenance
  • 05:53 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db2169.codfw.wmnet with reason: Maintenance
  • 05:53 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db2168.codfw.wmnet with reason: Maintenance
  • 05:52 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1236.eqiad.wmnet with reason: Maintenance
  • 05:50 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1359.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 05:50 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2348.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 05:50 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1359.eqiad.wmnet
  • 05:50 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2348.codfw.wmnet
  • 05:50 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on dbstore1008.eqiad.wmnet with reason: Maintenance
  • 05:49 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1359.eqiad.wmnet
  • 05:49 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2348.codfw.wmnet
  • 05:49 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1288.eqiad.wmnet with reason: Maintenance
  • 05:49 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1358.eqiad.wmnet
  • 05:49 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1358.eqiad.wmnet
  • 05:49 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2347.codfw.wmnet
  • 05:49 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1285.eqiad.wmnet with reason: Maintenance
  • 05:49 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1284.eqiad.wmnet with reason: Maintenance
  • 05:49 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2347.codfw.wmnet
  • 05:48 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 7 hosts with reason: Maintenance
  • 05:48 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2347.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 05:48 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1273.eqiad.wmnet with reason: Maintenance
  • 05:48 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1253.eqiad.wmnet with reason: Maintenance
  • 05:48 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1358.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 05:47 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1231.eqiad.wmnet with reason: Maintenance
  • 05:47 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1227.eqiad.wmnet with reason: Maintenance
  • 05:47 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1202.eqiad.wmnet with reason: Maintenance
  • 05:46 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1194.eqiad.wmnet with reason: Maintenance
  • 05:46 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1191.eqiad.wmnet with reason: Maintenance
  • 05:46 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1190.eqiad.wmnet with reason: Maintenance
  • 05:45 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1181.eqiad.wmnet with reason: Maintenance
  • 05:45 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1174.eqiad.wmnet with reason: Maintenance
  • 05:45 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1170.eqiad.wmnet with reason: Maintenance
  • 05:44 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1158.eqiad.wmnet with reason: Maintenance
  • 05:44 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2159.codfw.wmnet with reason: Maintenance
  • 05:42 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2228.codfw.wmnet with reason: Maintenance
  • 05:42 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2222.codfw.wmnet with reason: Maintenance
  • 05:42 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2347.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 05:41 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2221.codfw.wmnet with reason: Maintenance
  • 05:41 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2347.codfw.wmnet
  • 05:41 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2220.codfw.wmnet with reason: Maintenance
  • 05:41 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1358.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 05:41 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2347.codfw.wmnet
  • 05:41 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1358.eqiad.wmnet
  • 05:41 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2218.codfw.wmnet with reason: Maintenance
  • 05:41 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2346.codfw.wmnet
  • 05:41 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2346.codfw.wmnet
  • 05:40 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2208.codfw.wmnet with reason: Maintenance
  • 05:40 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1358.eqiad.wmnet
  • 05:40 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2200.codfw.wmnet with reason: Maintenance
  • 05:40 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1357.eqiad.wmnet
  • 05:40 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1357.eqiad.wmnet
  • 05:39 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2198.codfw.wmnet with reason: Maintenance
  • 05:39 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2182.codfw.wmnet with reason: Maintenance
  • 05:39 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2169.codfw.wmnet with reason: Maintenance
  • 05:38 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2346.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 05:38 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2168.codfw.wmnet with reason: Maintenance
  • 05:36 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db1162.eqiad.wmnet with reason: Maintenance
  • 05:32 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2346.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 05:31 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2346.codfw.wmnet
  • 05:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2346.codfw.wmnet
  • 05:31 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2345.codfw.wmnet
  • 05:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2345.codfw.wmnet
  • 05:30 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2345.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 05:29 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1357.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 05:24 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2345.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 05:23 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2345.codfw.wmnet
  • 05:23 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2345.codfw.wmnet
  • 05:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2344.codfw.wmnet
  • 05:22 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2344.codfw.wmnet
  • 05:22 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1357.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 05:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1357.eqiad.wmnet
  • 05:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1357.eqiad.wmnet
  • 05:21 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1356.eqiad.wmnet
  • 05:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1356.eqiad.wmnet
  • 05:11 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2344.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 05:05 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2344.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 05:04 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2344.codfw.wmnet
  • 05:04 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2344.codfw.wmnet
  • 05:04 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2343.codfw.wmnet
  • 05:04 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2343.codfw.wmnet
  • 05:03 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2343.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 04:56 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2343.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 04:56 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2343.codfw.wmnet
  • 04:56 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2343.codfw.wmnet
  • 04:55 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2342.codfw.wmnet
  • 04:55 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2342.codfw.wmnet
  • 04:41 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1356.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 04:35 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1356.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 04:34 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1356.eqiad.wmnet
  • 04:34 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1356.eqiad.wmnet
  • 04:34 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1355.eqiad.wmnet
  • 04:34 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1355.eqiad.wmnet
  • 04:33 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1355.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 04:26 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1355.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 04:26 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1355.eqiad.wmnet
  • 04:26 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1355.eqiad.wmnet
  • 04:25 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1354.eqiad.wmnet
  • 04:25 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1354.eqiad.wmnet
  • 04:10 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2342.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 04:10 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1354.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 04:03 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2342.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 04:03 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1354.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 04:03 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2342.codfw.wmnet
  • 04:03 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1354.eqiad.wmnet
  • 04:02 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2342.codfw.wmnet
  • 04:02 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1354.eqiad.wmnet
  • 04:02 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2341.codfw.wmnet
  • 04:02 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2341.codfw.wmnet
  • 04:02 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1353.eqiad.wmnet
  • 04:02 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1353.eqiad.wmnet
  • 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 38s)
  • 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image
  • 00:00 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
  • 00:00 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
  • 00:00 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
  • 00:00 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: sync

2026-10-15

  • 23:42 kemayo@deploy1003: Finished scap sync-world: Backport for EditCheck: Use binary search to match annotations to modified ranges (1.47.0-wmf.24) (duration: 10m 29s)
  • 23:37 kemayo@deploy1003: kemayo: Continuing with deployment
  • 23:36 kemayo@deploy1003: kemayo: Backport for EditCheck: Use binary search to match annotations to modified ranges (1.47.0-wmf.24) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 23:32 kemayo@deploy1003: Started scap sync-world: Backport for EditCheck: Use binary search to match annotations to modified ranges (1.47.0-wmf.24)
  • 23:24 zabe@deploy1003: Finished scap sync-world: Backport for Update interwiki cache (T441169) (duration: 13m 02s)
  • 23:20 pt1979@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['cp2047']
  • 23:19 zabe@deploy1003: zabe: Continuing with deployment
  • 23:15 zabe@deploy1003: zabe: Backport for Update interwiki cache (T441169) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 23:11 zabe@deploy1003: Started scap sync-world: Backport for Update interwiki cache (T441169)
  • 23:08 sbassett: Deployed security fix for T441037
  • 23:03 pt1979@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cp2047']
  • 22:51 sbassett: Deployed security fix for T440713
  • 22:37 sbassett: Deployed security fix for T439859
  • 22:24 sbassett: Deployed security fix for T439935
  • 22:08 sbassett: Deployed security patch for T437944
  • 21:58 ssastry@deploy1003: mwscript-k8s job started: ./extensions/PageImages/maintenance/initImageData.php --wiki=frwiki --queue-pressure=20 --later-than=20261015081300 # T441345: TAKE 3: frwiki: run initImageData for pages edited after .24 deploy and before fix
  • 21:44 maryum: Deployed security fix for T439722, T440123, and T440119
  • 21:34 ssastry@deploy1003: mwscript-k8s job started: ./extensions/PageImages/maintenance/initImageData.php --wiki=frwiki --queue-pressure=20 --later-than=20261015081300 --earlier-than=20261015144032 # T441345: TAKE 2: frwiki: run initImageData for pages edited after .24 deploy and before fix
  • 21:33 ssastry@deploy1003: mwscript-k8s job started: ./extensions/PageImages/maintenance/initImageData.php --wiki=frwiki --start=36374 --queue-pressure=20 --later-than=20261015081300 --earlier-than=20261015144032 # T441345: TAKE 2: frwiki: run initImageData for pages edited after .24 deploy and before fix
  • 21:31 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1353.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 21:31 cscott@deploy1003: Finished scap sync-world: Backport for Ensure that RefreshLinks::fixLinksFromArticle() runs secondary data updates (T441345) (1.47.0-wmf.24) (duration: 08m 11s)
  • 21:25 cscott@deploy1003: cscott: Continuing with deployment
  • 21:25 cscott@deploy1003: cscott: Backport for Ensure that RefreshLinks::fixLinksFromArticle() runs secondary data updates (T441345) (1.47.0-wmf.24) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 21:24 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1353.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 21:24 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1353.eqiad.wmnet
  • 21:23 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1353.eqiad.wmnet
  • 21:23 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1352.eqiad.wmnet
  • 21:23 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1352.eqiad.wmnet
  • 21:22 cscott@deploy1003: Started scap sync-world: Backport for Ensure that RefreshLinks::fixLinksFromArticle() runs secondary data updates (T441345) (1.47.0-wmf.24)
  • 21:15 kemayo@deploy1003: Finished scap sync-world: Backport for Suggestion mode: tighten behavior of 'view more' button in publish card (T432622) (1.47.0-wmf.24) (duration: 09m 20s)
  • 21:10 kemayo@deploy1003: kemayo: Continuing with deployment
  • 21:08 kemayo@deploy1003: kemayo: Backport for Suggestion mode: tighten behavior of 'view more' button in publish card (T432622) (1.47.0-wmf.24) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 21:08 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1352.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 21:06 kemayo@deploy1003: Started scap sync-world: Backport for Suggestion mode: tighten behavior of 'view more' button in publish card (T432622) (1.47.0-wmf.24)
  • 21:01 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1352.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 21:01 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp2047.codfw.wmnet
  • 21:01 cdobbins@cumin1004: START - Cookbook sre.hosts.remove-downtime for cp2047.codfw.wmnet
  • 21:01 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=cp2047.*
  • 21:01 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1352.eqiad.wmnet
  • 21:01 cdobbins@cumin1004: END (ERROR) - Cookbook sre.hardware.upgrade-firmware (exit_code=97) upgrade firmware for hosts ['cp2047']
  • 21:00 cscott@deploy1003: Finished scap sync-world: Backport for Hand-patch forward-compatibility with Parsoid's new page bundle format (T441221) (1.47.0-wmf.23) (duration: 06m 53s)
  • 21:00 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1352.eqiad.wmnet
  • 21:00 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1351.eqiad.wmnet
  • 20:59 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1351.eqiad.wmnet
  • 20:59 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1351.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 20:59 cscott@deploy1003: cscott: Continuing with deployment
  • 20:55 cscott@deploy1003: cscott: Backport for Hand-patch forward-compatibility with Parsoid's new page bundle format (T441221) (1.47.0-wmf.23) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 20:54 cscott@deploy1003: Started scap sync-world: Backport for Hand-patch forward-compatibility with Parsoid's new page bundle format (T441221) (1.47.0-wmf.23)
  • 20:52 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1351.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 20:52 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1351.eqiad.wmnet
  • 20:52 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1351.eqiad.wmnet
  • 20:51 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1350.eqiad.wmnet
  • 20:51 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1350.eqiad.wmnet
  • 20:51 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1350.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 20:51 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2341.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 20:48 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply
  • 20:44 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1350.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 20:44 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2341.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 20:44 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1350.eqiad.wmnet
  • 20:44 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2341.codfw.wmnet
  • 20:43 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1350.eqiad.wmnet
  • 20:43 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D{wikikube-worker[1350-1379].eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad)
  • 20:43 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2341.codfw.wmnet
  • 20:43 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D{wikikube-worker[2341-2364].codfw.wmnet} and (A:wikikube-master-codfw or A:wikikube-worker-codfw)
  • 20:35 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.enable_virtualization (exit_code=0) rolling provision on D{wikikube-worker[1330-1349].eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad)
  • 20:35 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1349.eqiad.wmnet
  • 20:35 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1349.eqiad.wmnet
  • 20:35 cdobbins@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cp2047']
  • 20:35 cdobbins@cumin1004: conftool action : set/pooled=no; selector: name=cp2047.*
  • 20:34 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=cp2049.*
  • 20:34 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp2049.codfw.wmnet
  • 20:34 cdobbins@cumin1004: START - Cookbook sre.hosts.remove-downtime for cp2049.codfw.wmnet
  • 20:32 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp2049.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 20:19 cdobbins@cumin1004: START - Cookbook sre.hosts.provision for host cp2049.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 20:17 kemayo@deploy1003: Finished scap sync-world: Backport for VisualEditor: Precompute edit suggestion counts on 50% of enwiki (T433024), InitialiseSettings: enable return-to-app banner on all wikis (T439746) (duration: 14m 43s)
  • 20:16 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply
  • 20:12 kemayo@deploy1003: kemayo, cklimas: Continuing with deployment
  • 20:11 cdobbins@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts cp2049.codfw.wmnet
  • 20:11 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cp2049.codfw.wmnet
  • 20:06 kemayo@deploy1003: kemayo, cklimas: Backport for VisualEditor: Precompute edit suggestion counts on 50% of enwiki (T433024), InitialiseSettings: enable return-to-app banner on all wikis (T439746) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 20:02 kemayo@deploy1003: Started scap sync-world: Backport for VisualEditor: Precompute edit suggestion counts on 50% of enwiki (T433024), InitialiseSettings: enable return-to-app banner on all wikis (T439746)
  • 19:48 cdobbins@cumin1004: START - Cookbook sre.hosts.reboot-single for host cp2049.codfw.wmnet
  • 19:46 cdobbins@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cp2049.codfw.wmnet
  • 19:46 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply
  • 19:46 cdobbins@cumin1004: END (ERROR) - Cookbook sre.hardware.upgrade-firmware (exit_code=97) upgrade firmware for hosts cp2049.codfw.wmnet
  • 19:44 kemayo@deploy1003: helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply
  • 19:43 kemayo@deploy1003: helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply
  • 19:42 kemayo@deploy1003: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply
  • 19:42 kemayo@deploy1003: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply
  • 19:41 kemayo@deploy1003: helmfile [staging] DONE helmfile.d/services/changeprop-jobqueue: apply
  • 19:41 kemayo@deploy1003: helmfile [staging] START helmfile.d/services/changeprop-jobqueue: apply
  • 19:40 kemayo@deploy1003: helmfile [eqiad] DONE helmfile.d/services/editcheck-headless: apply
  • 19:40 kemayo@deploy1003: helmfile [eqiad] START helmfile.d/services/editcheck-headless: apply
  • 19:39 kemayo@deploy1003: helmfile [codfw] DONE helmfile.d/services/editcheck-headless: apply
  • 19:38 kemayo@deploy1003: helmfile [codfw] START helmfile.d/services/editcheck-headless: apply
  • 19:37 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest1006.eqiad.wmnet with OS trixie
  • 19:24 ssastry@deploy1003: mwscript-k8s job started: ./extensions/PageImages/maintenance/initImageData.php --wiki=frwiki --start=36374 --queue-pressure=20 --later-than=20261015081300 --earlier-than=20261015144032 # T441345: run initImageData for pages edited after .24 deploy and before fix
  • 19:23 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply
  • 19:16 cdobbins@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cp2049.codfw.wmnet
  • 19:16 cdobbins@cumin1004: END (ERROR) - Cookbook sre.hardware.upgrade-firmware (exit_code=97) upgrade firmware for hosts cp2049.codfw.wmnet
  • 19:05 ladsgroup@deploy1003: Finished scap sync-world: Backport for Continuation of changes after the undeployment of FR on enwikibooks (T440519) (duration: 13m 12s)
  • 19:01 cmooney@cumin1004: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on sretest1006.eqiad.wmnet with reason: host reimage
  • 19:00 cmooney@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest1006.eqiad.wmnet with reason: host reimage
  • 19:00 ssastry@deploy1003: mwscript-k8s job started: ./extensions/PageImages/maintenance/initImageData.php --wiki=frwiki --later-than=20261015081300 --earlier-than=20261015144032 # T441345: run initImageData for pages edited after .24 deploy and before fix
  • 19:00 ladsgroup@deploy1003: codenamenoreste, ladsgroup: Continuing with deployment
  • 18:58 ssastry@deploy1003: mwscript-k8s job started: initImageData.php --wiki=frwiki --later-than=20261015081300 --earlier-than=20261015144032 # T441345: run initImageData for pages edited after .24 deploy and before fix
  • 18:58 ssastry@deploy1003: mwscript-k8s job started: initImageData.php --later-than=20261015081300 --earlier-than=20261015144032 --wiki=frwiki # T441345: run initImageData for pages edited after .24 deploy and before fix
  • 18:56 ladsgroup@deploy1003: codenamenoreste, ladsgroup: Backport for Continuation of changes after the undeployment of FR on enwikibooks (T440519) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 18:56 cdobbins@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cp2049.codfw.wmnet
  • 18:55 cdobbins@cumin1004: conftool action : set/pooled=no; selector: name=cp2049.*
  • 18:54 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=cp2048.*
  • 18:54 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp2048.codfw.wmnet
  • 18:54 cdobbins@cumin1004: START - Cookbook sre.hosts.remove-downtime for cp2048.codfw.wmnet
  • 18:52 ladsgroup@deploy1003: Started scap sync-world: Backport for Continuation of changes after the undeployment of FR on enwikibooks (T440519)
  • 18:47 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=cp2047.*
  • 18:47 cdobbins@cumin1004: END (ERROR) - Cookbook sre.hardware.upgrade-firmware (exit_code=97) upgrade firmware for hosts ['cp2047']
  • 18:46 cdobbins@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cp2047']
  • 18:46 cdobbins@cumin1004: conftool action : set/pooled=no; selector: name=cp2047.*
  • 18:43 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp2048.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 18:42 cmooney@cumin1004: START - Cookbook sre.hosts.reimage for host sretest1006.eqiad.wmnet with OS trixie
  • 18:42 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=cp2047.*
  • 18:42 cdobbins@cumin1004: END (ERROR) - Cookbook sre.hardware.upgrade-firmware (exit_code=97) upgrade firmware for hosts ['cp2047']
  • 18:41 cdobbins@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cp2047']
  • 18:41 cdobbins@cumin1004: conftool action : set/pooled=no; selector: name=cp2047.*
  • 18:41 cdobbins@cumin1004: END (ERROR) - Cookbook sre.hardware.upgrade-firmware (exit_code=97) upgrade firmware for hosts ['cp2047']
  • 18:40 cdobbins@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cp2047']
  • 18:39 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) sretest1006.eqiad.wmnet on all recursors
  • 18:39 cmooney@cumin1004: START - Cookbook sre.dns.wipe-cache sretest1006.eqiad.wmnet on all recursors
  • 18:39 musikanimal@deploy1003: Finished scap sync-world: Backport for CommonSettings: keep CodeMirror off on enwiki for older accounts (T441308) (duration: 10m 14s)
  • 18:38 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 18:38 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update entry for sretest1006 - cmooney@cumin1004"
  • 18:37 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update entry for sretest1006 - cmooney@cumin1004"
  • 18:37 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.enable_virtualization (exit_code=0) rolling provision on D{wikikube-worker[2321-2340].codfw.wmnet} and (A:wikikube-master-codfw or A:wikikube-worker-codfw)
  • 18:37 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2340.codfw.wmnet
  • 18:37 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2340.codfw.wmnet
  • 18:37 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2340.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 18:33 cmooney@cumin1004: START - Cookbook sre.dns.netbox
  • 18:33 musikanimal@deploy1003: musikanimal: Continuing with deployment
  • 18:32 musikanimal@deploy1003: musikanimal: Backport for CommonSettings: keep CodeMirror off on enwiki for older accounts (T441308) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 18:31 cdobbins@cumin1004: START - Cookbook sre.hosts.provision for host cp2048.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 18:30 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2340.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 18:30 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2340.codfw.wmnet
  • 18:29 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2340.codfw.wmnet
  • 18:29 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2339.codfw.wmnet
  • 18:29 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2339.codfw.wmnet
  • 18:29 musikanimal@deploy1003: Started scap sync-world: Backport for CommonSettings: keep CodeMirror off on enwiki for older accounts (T441308)
  • 18:27 cdobbins@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts cp2048.codfw.wmnet
  • 18:27 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cp2048.codfw.wmnet
  • 18:25 pt1979@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cp2047']
  • 18:23 pt1979@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['cp2047']
  • 18:22 pt1979@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cp2047']
  • 18:19 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply
  • 18:18 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2339.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 18:11 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2339.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 18:11 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2339.codfw.wmnet
  • 18:10 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2339.codfw.wmnet
  • 18:10 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2338.codfw.wmnet
  • 18:10 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2338.codfw.wmnet
  • 18:09 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2338.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 18:05 cdobbins@cumin1004: START - Cookbook sre.hosts.reboot-single for host cp2048.codfw.wmnet
  • 18:04 cdobbins@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cp2048.codfw.wmnet
  • 18:04 cdobbins@cumin1004: conftool action : set/pooled=no; selector: name=cp2048.*
  • 18:03 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2338.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 18:02 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2338.codfw.wmnet
  • 18:02 cmooney@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest1006.eqiad.wmnet with OS trixie
  • 18:02 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2338.codfw.wmnet
  • 18:01 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2337.codfw.wmnet
  • 18:01 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2337.codfw.wmnet
  • 18:01 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2337.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 17:54 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2337.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 17:54 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2337.codfw.wmnet
  • 17:54 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2337.codfw.wmnet
  • 17:53 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2336.codfw.wmnet
  • 17:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2336.codfw.wmnet
  • 17:43 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1349.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 17:43 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=cp2047.*
  • 17:42 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp2047.codfw.wmnet
  • 17:42 cdobbins@cumin1004: START - Cookbook sre.hosts.remove-downtime for cp2047.codfw.wmnet
  • 17:41 dancy@deploy1003: Installation of scap version "4.299.0" completed for 3 hosts
  • 17:39 dancy@deploy1003: Installing scap version "4.299.0" for 3 host(s)
  • 17:39 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp2047.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 17:38 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2336.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 17:37 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1349.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 17:36 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1349.eqiad.wmnet
  • 17:36 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1349.eqiad.wmnet
  • 17:36 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1348.eqiad.wmnet
  • 17:35 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1348.eqiad.wmnet
  • 17:33 tsev@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply
  • 17:32 tsev@deploy1003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply
  • 17:31 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2336.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 17:31 tsev@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply
  • 17:31 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2336.codfw.wmnet
  • 17:31 tsev@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply
  • 17:30 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2336.codfw.wmnet
  • 17:30 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2335.codfw.wmnet
  • 17:30 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2335.codfw.wmnet
  • 17:27 tsev@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply
  • 17:27 tsev@deploy1003: helmfile [staging] START helmfile.d/services/wikifeeds: apply
  • 17:23 cdobbins@cumin1004: START - Cookbook sre.hosts.provision for host cp2047.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 17:20 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1348.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 17:19 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2335.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 17:13 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1348.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 17:13 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1348.eqiad.wmnet
  • 17:12 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1348.eqiad.wmnet
  • 17:12 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1347.eqiad.wmnet
  • 17:12 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1347.eqiad.wmnet
  • 17:12 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2335.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 17:12 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2335.codfw.wmnet
  • 17:11 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2335.codfw.wmnet
  • 17:11 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2334.codfw.wmnet
  • 17:11 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2334.codfw.wmnet
  • 17:11 cdobbins@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts cp2047.codfw.wmnet
  • 17:11 cdobbins@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cp2047.codfw.wmnet
  • 17:10 cdobbins@cumin1004: END (ERROR) - Cookbook sre.hardware.upgrade-firmware (exit_code=97) upgrade firmware for hosts cp2047.codfw.wmnet
  • 17:05 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1347.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 17:02 arlolra@deploy1003: Finished scap sync-world: Backport for Set wgUseParsoidParser to false in prod (T441345 T432760) (duration: 08m 01s)
  • 17:00 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2334.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 16:58 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1347.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 16:58 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1347.eqiad.wmnet
  • 16:58 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1347.eqiad.wmnet
  • 16:57 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1346.eqiad.wmnet
  • 16:57 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1346.eqiad.wmnet
  • 16:56 arlolra@deploy1003: arlolra: Continuing with deployment
  • 16:56 arlolra@deploy1003: arlolra: Backport for Set wgUseParsoidParser to false in prod (T441345 T432760) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 16:56 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1346.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 16:54 arlolra@deploy1003: Started scap sync-world: Backport for Set wgUseParsoidParser to false in prod (T441345 T432760)
  • 16:53 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2334.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 16:53 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2334.codfw.wmnet
  • 16:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2334.codfw.wmnet
  • 16:52 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2333.codfw.wmnet
  • 16:52 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2333.codfw.wmnet
  • 16:51 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2333.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 16:49 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1346.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 16:49 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1346.eqiad.wmnet
  • 16:48 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1346.eqiad.wmnet
  • 16:48 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1345.eqiad.wmnet
  • 16:48 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1345.eqiad.wmnet
  • 16:46 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1345.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 16:45 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2333.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 16:44 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2333.codfw.wmnet
  • 16:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2333.codfw.wmnet
  • 16:44 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2332.codfw.wmnet
  • 16:43 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2332.codfw.wmnet
  • 16:43 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2332.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 16:40 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1345.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 16:39 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1345.eqiad.wmnet
  • 16:36 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2332.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 16:36 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2332.codfw.wmnet
  • 16:35 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2332.codfw.wmnet
  • 16:35 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2331.codfw.wmnet
  • 16:34 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1345.eqiad.wmnet
  • 16:34 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2331.codfw.wmnet
  • 16:34 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1344.eqiad.wmnet
  • 16:34 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1344.eqiad.wmnet
  • 16:34 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2331.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 16:29 cmooney@cumin1004: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on sretest1006.eqiad.wmnet with reason: host reimage
  • 16:29 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2331.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 16:28 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2331.codfw.wmnet
  • 16:28 cmooney@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest1006.eqiad.wmnet with reason: host reimage
  • 16:28 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2331.codfw.wmnet
  • 16:28 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2330.codfw.wmnet
  • 16:26 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2330.codfw.wmnet
  • 16:26 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2330.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 16:24 mutante: contint1003/contint2003 - restart docker to apply a nofile ulimit for containers from gerrit:1355127 T440670
  • 16:23 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1344.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 16:21 pt1979@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:05:00 on 8 hosts with reason: Testing PDU downtime
  • 16:21 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2330.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 16:20 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2330.codfw.wmnet
  • 16:20 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2330.codfw.wmnet
  • 16:20 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2329.codfw.wmnet
  • 16:19 pt1979@cumin2003: DONE (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 0:05:00 on 42 hosts with reason: Testing PDU downtime
  • 16:18 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2329.codfw.wmnet
  • 16:18 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2329.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 16:17 mutante: contint1002/contint2002 - restart docker to apply a nofile ulimit for containers from gerrit:1355127 T440670
  • 16:17 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1344.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 16:16 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1344.eqiad.wmnet
  • 16:16 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1344.eqiad.wmnet
  • 16:16 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1343.eqiad.wmnet
  • 16:15 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1343.eqiad.wmnet
  • 16:15 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
  • 16:14 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db1276.eqiad.wmnet with reason: Maintenance
  • 16:14 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on 7 hosts with reason: Maintenance
  • 16:14 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db1271.eqiad.wmnet with reason: Maintenance
  • 16:13 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db1259.eqiad.wmnet with reason: Maintenance
  • 16:13 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db1254.eqiad.wmnet with reason: Maintenance
  • 16:13 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2329.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 16:13 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db1239.eqiad.wmnet with reason: Maintenance
  • 16:13 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db1233.eqiad.wmnet with reason: Maintenance
  • 16:13 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2329.codfw.wmnet
  • 16:12 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db1229.eqiad.wmnet with reason: Maintenance
  • 16:12 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2329.codfw.wmnet
  • 16:12 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db1225.eqiad.wmnet with reason: Maintenance
  • 16:12 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2328.codfw.wmnet
  • 16:11 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2328.codfw.wmnet
  • 16:11 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db1222.eqiad.wmnet with reason: Maintenance
  • 16:11 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db1197.eqiad.wmnet with reason: Maintenance
  • 16:11 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2328.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 16:11 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db1188.eqiad.wmnet with reason: Maintenance
  • 16:10 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db1156.eqiad.wmnet with reason: Maintenance
  • 16:10 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db2204.codfw.wmnet with reason: Maintenance
  • 16:06 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2328.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 16:05 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2328.codfw.wmnet
  • 16:05 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2328.codfw.wmnet
  • 16:05 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2327.codfw.wmnet
  • 16:05 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1343.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 16:04 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2327.codfw.wmnet
  • 16:03 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2327.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 15:58 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2327.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 15:58 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1343.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 15:57 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db2238.codfw.wmnet with reason: Maintenance
  • 15:56 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db2226.codfw.wmnet with reason: Maintenance
  • 15:56 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db2225.codfw.wmnet with reason: Maintenance
  • 15:55 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db2207.codfw.wmnet with reason: Maintenance
  • 15:55 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db2197.codfw.wmnet with reason: Maintenance
  • 15:54 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db2189.codfw.wmnet with reason: Maintenance
  • 15:54 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db2175.codfw.wmnet with reason: Maintenance
  • 15:52 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db1173.eqiad.wmnet with reason: Maintenance
  • 15:52 ebernhardson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search: apply
  • 15:52 ebernhardson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-semantic-search: apply
  • 15:49 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search: apply
  • 15:49 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search: apply
  • 15:40 cmooney@cumin1004: START - Cookbook sre.hosts.reimage for host sretest1006.eqiad.wmnet with OS trixie
  • 15:33 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cp2047.codfw.wmnet
  • 15:28 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "sync switches in eqiad a and b changed to state active in netbox - cmooney@cumin1004"
  • 15:28 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "sync switches in eqiad a and b changed to state active in netbox - cmooney@cumin1004"
  • 15:21 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1343.eqiad.wmnet
  • 15:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1343.eqiad.wmnet
  • 15:21 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1342.eqiad.wmnet
  • 15:20 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1342.eqiad.wmnet
  • 15:19 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1342.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 15:19 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/analytics-test: apply
  • 15:19 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/analytics-test: apply
  • 15:15 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2327.codfw.wmnet
  • 15:15 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2327.codfw.wmnet
  • 15:14 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2326.codfw.wmnet
  • 15:14 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2326.codfw.wmnet
  • 15:14 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2326.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 15:12 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1342.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 15:12 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1342.eqiad.wmnet
  • 15:12 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1342.eqiad.wmnet
  • 15:11 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1341.eqiad.wmnet
  • 15:11 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1341.eqiad.wmnet
  • 15:11 cdobbins@cumin1004: START - Cookbook sre.hosts.reboot-single for host cp2047.codfw.wmnet
  • 15:10 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1341.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 15:09 cdobbins@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cp2047.codfw.wmnet
  • 15:09 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 14 hosts with reason: Junos upgrade
  • 15:09 cdobbins@cumin1004: conftool action : set/pooled=no; selector: name=cp2047.*
  • 15:09 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2326.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 15:08 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2326.codfw.wmnet
  • 15:08 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2326.codfw.wmnet
  • 15:07 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2325.codfw.wmnet
  • 15:07 pt1979@cumin1004: DONE (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on 15 hosts with reason: Junos upgrade
  • 15:06 pt1979@cumin1004: DONE (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on 15 hosts with reason: Junos upgrade
  • 15:06 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2325.codfw.wmnet
  • 15:06 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2325.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 15:03 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1341.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 15:03 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1341.eqiad.wmnet
  • 15:02 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1341.eqiad.wmnet
  • 15:02 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1340.eqiad.wmnet
  • 15:02 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1340.eqiad.wmnet
  • 15:02 pt1979@cumin1004: DONE (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on 33 hosts with reason: Junos upgrade
  • 15:01 moritzm: installing unbound security updates
  • 15:00 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2325.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 15:00 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2325.codfw.wmnet
  • 15:00 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2325.codfw.wmnet
  • 14:59 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2324.codfw.wmnet
  • 14:59 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2324.codfw.wmnet
  • 14:58 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2324.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 14:57 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on dbstore1009.eqiad.wmnet with reason: Maintenance
  • 14:56 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db1287.eqiad.wmnet with reason: Maintenance
  • 14:56 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db1282.eqiad.wmnet with reason: Maintenance
  • 14:56 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on 6 hosts with reason: Maintenance
  • 14:55 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db1279.eqiad.wmnet with reason: Maintenance
  • 14:55 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db1241.eqiad.wmnet with reason: Maintenance
  • 14:55 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db1225.eqiad.wmnet with reason: Maintenance
  • 14:55 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db1201.eqiad.wmnet with reason: Maintenance
  • 14:54 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db1187.eqiad.wmnet with reason: Maintenance
  • 14:54 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db1168.eqiad.wmnet with reason: Maintenance
  • 14:54 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db1165.eqiad.wmnet with reason: Maintenance
  • 14:53 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2324.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 14:53 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2324.codfw.wmnet
  • 14:52 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2324.codfw.wmnet
  • 14:52 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2323.codfw.wmnet
  • 14:51 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db2229.codfw.wmnet with reason: Maintenance
  • 14:51 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2323.codfw.wmnet
  • 14:51 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2323.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 14:51 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2224 (T441197)', diff saved to https://phabricator.wikimedia.org/P96713 and previous config saved to /var/cache/conftool/dbconfig/20261015-145134-marostegui.json
  • 14:50 papaul: start maintenance on mr1-codfw
  • 14:48 topranks: enable BGP peerings to new spine switches eqiad rows A/B
  • 14:46 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1340.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 14:46 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2323.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 14:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2323.codfw.wmnet
  • 14:45 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2323.codfw.wmnet
  • 14:45 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2322.codfw.wmnet
  • 14:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2322.codfw.wmnet
  • 14:44 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2322.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 14:43 XioNoX: uploaded spicerack_14.0.0 to apt.wikimedia.org bookworm-wikimedia,trixie-wikimedia
  • 14:40 ladsgroup@deploy1003: Finished scap sync-world: Backport for Bump APCu expiry from 10s to 60s (T440885) (1.47.0-wmf.24) (duration: 09m 57s)
  • 14:40 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1340.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 14:39 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1340.eqiad.wmnet
  • 14:39 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1340.eqiad.wmnet
  • 14:39 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1339.eqiad.wmnet
  • 14:39 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1339.eqiad.wmnet
  • 14:39 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2322.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 14:38 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2322.codfw.wmnet
  • 14:38 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2322.codfw.wmnet
  • 14:37 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2321.codfw.wmnet
  • 14:37 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1339.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 14:36 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2224', diff saved to https://phabricator.wikimedia.org/P96712 and previous config saved to /var/cache/conftool/dbconfig/20261015-143632-marostegui.json
  • 14:36 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2321.codfw.wmnet
  • 14:36 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2321.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 14:35 ladsgroup@deploy1003: ladsgroup, krinkle: Continuing with deployment
  • 14:34 ladsgroup@deploy1003: ladsgroup, krinkle: Backport for Bump APCu expiry from 10s to 60s (T440885) (1.47.0-wmf.24) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 14:31 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2321.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 14:30 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1339.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 14:30 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2321.codfw.wmnet
  • 14:30 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1339.eqiad.wmnet
  • 14:30 ladsgroup@deploy1003: Started scap sync-world: Backport for Bump APCu expiry from 10s to 60s (T440885) (1.47.0-wmf.24)
  • 14:30 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2321.codfw.wmnet
  • 14:30 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1339.eqiad.wmnet
  • 14:30 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D{wikikube-worker[2321-2340].codfw.wmnet} and (A:wikikube-master-codfw or A:wikikube-worker-codfw)
  • 14:29 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1338.eqiad.wmnet
  • 14:29 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1338.eqiad.wmnet
  • 14:29 ladsgroup@deploy1003: Finished scap sync-world: Backport for Undeploy FR from enwikibooks (T440519) (duration: 11m 32s)
  • 14:24 ladsgroup@deploy1003: ladsgroup: Continuing with deployment
  • 14:22 ladsgroup@deploy1003: ladsgroup: Backport for Undeploy FR from enwikibooks (T440519) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 14:21 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2224', diff saved to https://phabricator.wikimedia.org/P96711 and previous config saved to /var/cache/conftool/dbconfig/20261015-142130-marostegui.json
  • 14:18 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1338.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 14:18 ladsgroup@deploy1003: Started scap sync-world: Backport for Undeploy FR from enwikibooks (T440519)
  • 14:12 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1338.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 14:11 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1338.eqiad.wmnet
  • 14:11 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1338.eqiad.wmnet
  • 14:11 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1337.eqiad.wmnet
  • 14:11 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1337.eqiad.wmnet
  • 14:10 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-sre: apply
  • 14:09 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-sre: apply
  • 14:09 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1337.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 14:06 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2224 (T441197)', diff saved to https://phabricator.wikimedia.org/P96710 and previous config saved to /var/cache/conftool/dbconfig/20261015-140628-marostegui.json
  • 14:05 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-sre: apply
  • 14:05 marostegui@cumin1004: dbctl commit (dc=all): 'Depooling db2224 (T441197)', diff saved to https://phabricator.wikimedia.org/P96709 and previous config saved to /var/cache/conftool/dbconfig/20261015-140520-marostegui.json
  • 14:05 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db2224.codfw.wmnet with reason: Maintenance
  • 14:05 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2217 (T441197)', diff saved to https://phabricator.wikimedia.org/P96708 and previous config saved to /var/cache/conftool/dbconfig/20261015-140502-marostegui.json
  • 14:04 marostegui: Drop ce_worklist_articles from wikisharede testwiki test2wiki officewiki T441263
  • 14:04 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-sre: apply
  • 14:03 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1337.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 14:02 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1337.eqiad.wmnet
  • 14:02 lucaswerkmeister-wmde@deploy1003: mwscript-k8s job started: cleanupTitles ukwiki # T441045 (expecting no results but running just in case)
  • 14:02 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for Remove eduction namespaces from Ukrainian Wikipedia (T441045), Add rollbacker user group to krcwiki (T440898) (duration: 13m 35s)
  • 14:01 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1337.eqiad.wmnet
  • 14:01 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1336.eqiad.wmnet
  • 14:01 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1336.eqiad.wmnet
  • 13:58 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
  • 13:58 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
  • 13:56 lucaswerkmeister-wmde@deploy1003: codenamenoreste, lucaswerkmeister-wmde: Continuing with deployment
  • 13:52 lucaswerkmeister-wmde@deploy1003: codenamenoreste, lucaswerkmeister-wmde: Backport for Remove eduction namespaces from Ukrainian Wikipedia (T441045), Add rollbacker user group to krcwiki (T440898) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 13:50 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2217', diff saved to https://phabricator.wikimedia.org/P96707 and previous config saved to /var/cache/conftool/dbconfig/20261015-135000-marostegui.json
  • 13:48 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for Remove eduction namespaces from Ukrainian Wikipedia (T441045), Add rollbacker user group to krcwiki (T440898)
  • 13:45 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1336.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 13:41 moritzm: installing unbound security updates
  • 13:39 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1336.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 13:39 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1336.eqiad.wmnet
  • 13:38 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1336.eqiad.wmnet
  • 13:38 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1335.eqiad.wmnet
  • 13:38 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1335.eqiad.wmnet
  • 13:34 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2217', diff saved to https://phabricator.wikimedia.org/P96706 and previous config saved to /var/cache/conftool/dbconfig/20261015-133458-marostegui.json
  • 13:29 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.enable_virtualization (exit_code=0) rolling provision on D{wikikube-worker[2293-2320].codfw.wmnet} and (A:wikikube-master-codfw or A:wikikube-worker-codfw)
  • 13:29 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2320.codfw.wmnet
  • 13:29 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2320.codfw.wmnet
  • 13:29 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2320.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 13:27 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1335.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 13:26 isaranto@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
  • 13:23 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2320.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 13:23 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2320.codfw.wmnet
  • 13:22 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2320.codfw.wmnet
  • 13:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2319.codfw.wmnet
  • 13:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2319.codfw.wmnet
  • 13:21 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2319.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 13:20 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1335.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 13:20 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1335.eqiad.wmnet
  • 13:19 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2217 (T441197)', diff saved to https://phabricator.wikimedia.org/P96705 and previous config saved to /var/cache/conftool/dbconfig/20261015-131956-marostegui.json
  • 13:19 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1335.eqiad.wmnet
  • 13:19 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D{wikikube-worker[1330-1349].eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad)
  • 13:18 marostegui@cumin1004: dbctl commit (dc=all): 'Depooling db2217 (T441197)', diff saved to https://phabricator.wikimedia.org/P96704 and previous config saved to /var/cache/conftool/dbconfig/20261015-131848-marostegui.json
  • 13:18 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db2217.codfw.wmnet with reason: Maintenance
  • 13:18 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2214 (T441197)', diff saved to https://phabricator.wikimedia.org/P96703 and previous config saved to /var/cache/conftool/dbconfig/20261015-131831-marostegui.json
  • 13:15 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2319.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 13:15 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2319.codfw.wmnet
  • 13:15 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2319.codfw.wmnet
  • 13:14 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2318.codfw.wmnet
  • 13:13 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2318.codfw.wmnet
  • 13:13 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2318.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 13:07 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2318.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 13:07 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2318.codfw.wmnet
  • 13:06 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2318.codfw.wmnet
  • 13:06 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2317.codfw.wmnet
  • 13:05 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2317.codfw.wmnet
  • 13:05 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2317.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 13:03 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2214', diff saved to https://phabricator.wikimedia.org/P96702 and previous config saved to /var/cache/conftool/dbconfig/20261015-130328-marostegui.json
  • 13:00 jclark@cumin1004: START - Cookbook sre.hosts.reimage for host wikikube-worker1298.eqiad.wmnet with OS bookworm
  • 13:00 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2317.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 12:59 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2317.codfw.wmnet
  • 12:59 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2317.codfw.wmnet
  • 12:58 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2316.codfw.wmnet
  • 12:58 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2316.codfw.wmnet
  • 12:53 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1334.eqiad.wmnet
  • 12:52 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1334.eqiad.wmnet
  • 12:52 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1333.eqiad.wmnet
  • 12:52 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1333.eqiad.wmnet
  • 12:52 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1332.eqiad.wmnet
  • 12:52 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1332.eqiad.wmnet
  • 12:51 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1331.eqiad.wmnet
  • 12:51 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1331.eqiad.wmnet
  • 12:48 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2214', diff saved to https://phabricator.wikimedia.org/P96701 and previous config saved to /var/cache/conftool/dbconfig/20261015-124826-marostegui.json
  • 12:41 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'.
  • 12:40 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'.
  • 12:39 dpogorzelski@cumin1004: conftool action : set/pooled=true; selector: dnsdisc=inference,name=codfw
  • 12:39 dpogorzelski@cumin1004: conftool action : set/pooled=true; selector: dnsdisc=k8s-ingress-ml-serve,name=codfw
  • 12:38 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'.
  • 12:35 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'.
  • 12:35 dpogorzelski@cumin1004: conftool action : set/pooled=false; selector: dnsdisc=k8s-ingress-ml-serve,name=codfw
  • 12:35 dpogorzelski@cumin1004: conftool action : set/pooled=false; selector: dnsdisc=inference,name=codfw
  • 12:35 dpogorzelski@cumin1004: conftool action : set/pooled=true; selector: dnsdisc=inference,name=eqiad
  • 12:35 dpogorzelski@cumin1004: conftool action : set/pooled=true; selector: dnsdisc=k8s-ingress-ml-serve,name=eqiad
  • 12:34 dpogorzelski@cumin1004: conftool action : set/pooled=false; selector: dnsdisc=k8s-ingress-ml-serve,name=eqiad
  • 12:34 dpogorzelski@cumin1004: conftool action : set/pooled=false; selector: dnsdisc=inference,name=eqiad
  • 12:33 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2214 (T441197)', diff saved to https://phabricator.wikimedia.org/P96700 and previous config saved to /var/cache/conftool/dbconfig/20261015-123325-marostegui.json
  • 12:31 marostegui@cumin1004: dbctl commit (dc=all): 'Depooling db2214 (T441197)', diff saved to https://phabricator.wikimedia.org/P96699 and previous config saved to /var/cache/conftool/dbconfig/20261015-123116-marostegui.json
  • 12:31 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db2214.codfw.wmnet with reason: Maintenance
  • 12:31 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db2197.codfw.wmnet with reason: Maintenance
  • 12:30 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2193 (T441197)', diff saved to https://phabricator.wikimedia.org/P96698 and previous config saved to /var/cache/conftool/dbconfig/20261015-123048-marostegui.json
  • 12:18 topranks: reboot ssw1-d8-eqiad to upgrade SR-Linux on it T431745
  • 12:15 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2193', diff saved to https://phabricator.wikimedia.org/P96697 and previous config saved to /var/cache/conftool/dbconfig/20261015-121546-marostegui.json
  • 12:09 cmooney@cumin1004: DONE (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 1:00:00 on lsw1-c1-eqiad with reason: upgrade ssw1-d8-eqiad
  • 12:08 cmooney@cumin1004: DONE (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 1:00:00 on ssw1-c1-eqiad with reason: upgrade ssw1-d8-eqiad
  • 12:08 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 19 hosts with reason: upgrade ssw1-d8-eqiad
  • 12:00 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2193', diff saved to https://phabricator.wikimedia.org/P96696 and previous config saved to /var/cache/conftool/dbconfig/20261015-120044-marostegui.json
  • 11:56 topranks: drain traffic from ssw1-d8-eqiad in advance of SR-Linux upgrade T431745
  • 11:47 kart_: Updated Recommendation API to 2026-10-07-145851-production (staging only) (T437213)
  • 11:45 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2193 (T441197)', diff saved to https://phabricator.wikimedia.org/P96695 and previous config saved to /var/cache/conftool/dbconfig/20261015-114543-marostegui.json
  • 11:44 kartik@deploy1003: helmfile [ml-staging-codfw] 'sync' command on namespace 'recommendation-api-ng' for release 'main' .
  • 11:43 marostegui@cumin1004: dbctl commit (dc=all): 'Depooling db2193 (T441197)', diff saved to https://phabricator.wikimedia.org/P96694 and previous config saved to /var/cache/conftool/dbconfig/20261015-114335-marostegui.json
  • 11:43 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db2193.codfw.wmnet with reason: Maintenance
  • 11:43 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2180 (T441197)', diff saved to https://phabricator.wikimedia.org/P96693 and previous config saved to /var/cache/conftool/dbconfig/20261015-114317-marostegui.json
  • 11:28 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2180', diff saved to https://phabricator.wikimedia.org/P96692 and previous config saved to /var/cache/conftool/dbconfig/20261015-112815-marostegui.json
  • 11:13 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2180', diff saved to https://phabricator.wikimedia.org/P96691 and previous config saved to /var/cache/conftool/dbconfig/20261015-111313-marostegui.json
  • 11:10 root@cumin2003: END (PASS) - Cookbook sre.misc-clusters.roll-restart-reboot-eventschemas (exit_code=0) rolling restart_daemons on A:schema
  • 11:07 root@cumin2003: START - Cookbook sre.misc-clusters.roll-restart-reboot-eventschemas rolling restart_daemons on A:schema
  • 10:58 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2180 (T441197)', diff saved to https://phabricator.wikimedia.org/P96690 and previous config saved to /var/cache/conftool/dbconfig/20261015-105811-marostegui.json
  • 10:56 marostegui@cumin1004: dbctl commit (dc=all): 'Depooling db2180 (T441197)', diff saved to https://phabricator.wikimedia.org/P96689 and previous config saved to /var/cache/conftool/dbconfig/20261015-105601-marostegui.json
  • 10:56 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db2180.codfw.wmnet with reason: Maintenance
  • 10:55 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2158 (T441197)', diff saved to https://phabricator.wikimedia.org/P96688 and previous config saved to /var/cache/conftool/dbconfig/20261015-105543-marostegui.json
  • 10:40 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2158', diff saved to https://phabricator.wikimedia.org/P96687 and previous config saved to /var/cache/conftool/dbconfig/20261015-104041-marostegui.json
  • 10:25 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2158', diff saved to https://phabricator.wikimedia.org/P96686 and previous config saved to /var/cache/conftool/dbconfig/20261015-102539-marostegui.json
  • 10:23 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
  • 10:11 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
  • 10:10 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
  • 10:10 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2158 (T441197)', diff saved to https://phabricator.wikimedia.org/P96685 and previous config saved to /var/cache/conftool/dbconfig/20261015-101037-marostegui.json
  • 10:08 marostegui@cumin1004: dbctl commit (dc=all): 'Depooling db2158 (T441197)', diff saved to https://phabricator.wikimedia.org/P96684 and previous config saved to /var/cache/conftool/dbconfig/20261015-100829-marostegui.json
  • 10:08 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db2158.codfw.wmnet with reason: Maintenance
  • 10:04 marostegui: deploy schema change on s8 with replication T441197
  • 10:03 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
  • 09:48 root@cumin2003: END (PASS) - Cookbook sre.o11y.roll-restart-reboot-logstash-collectors (exit_code=0) rolling restart_daemons on A:logstash-collector
  • 09:40 root@cumin2003: START - Cookbook sre.o11y.roll-restart-reboot-logstash-collectors rolling restart_daemons on A:logstash-collector
  • 09:35 hashar@deploy1003: Finished scap sync-world: Backport for Discord: Pass array options to ParserOutputAccess::getParserOutput (T440951) (1.47.0-wmf.24) (duration: 09m 50s)
  • 09:29 hashar@deploy1003: jforrester, hashar: Continuing with deployment
  • 09:29 hashar@deploy1003: jforrester, hashar: Backport for Discord: Pass array options to ParserOutputAccess::getParserOutput (T440951) (1.47.0-wmf.24) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 09:28 moritzm: installing expat security updates
  • 09:25 hashar@deploy1003: Started scap sync-world: Backport for Discord: Pass array options to ParserOutputAccess::getParserOutput (T440951) (1.47.0-wmf.24)
  • 09:20 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2316.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 09:19 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthbook: apply
  • 09:19 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook: apply
  • 09:15 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2316.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 09:15 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2316.codfw.wmnet
  • 09:14 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2316.codfw.wmnet
  • 09:14 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2315.codfw.wmnet
  • 09:13 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2315.codfw.wmnet
  • 09:12 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2315.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 09:10 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
  • 09:07 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2315.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 09:07 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2315.codfw.wmnet
  • 09:06 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
  • 09:06 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2315.codfw.wmnet
  • 09:06 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2314.codfw.wmnet
  • 09:05 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'.
  • 09:04 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2314.codfw.wmnet
  • 09:04 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2314.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 09:03 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'.
  • 08:59 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2314.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 08:58 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2314.codfw.wmnet
  • 08:58 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2314.codfw.wmnet
  • 08:57 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2313.codfw.wmnet
  • 08:56 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2313.codfw.wmnet
  • 08:55 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2313.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 08:50 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2313.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 08:50 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2313.codfw.wmnet
  • 08:49 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2313.codfw.wmnet
  • 08:49 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2312.codfw.wmnet
  • 08:47 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2312.codfw.wmnet
  • 08:47 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2312.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 08:42 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2312.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 08:41 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2312.codfw.wmnet
  • 08:41 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
  • 08:40 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2312.codfw.wmnet
  • 08:40 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2311.codfw.wmnet
  • 08:40 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
  • 08:39 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2311.codfw.wmnet
  • 08:39 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2311.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 08:34 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2311.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 08:33 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2311.codfw.wmnet
  • 08:33 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2311.codfw.wmnet
  • 08:33 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2310.codfw.wmnet
  • 08:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2310.codfw.wmnet
  • 08:31 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2310.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 08:31 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
  • 08:29 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply
  • 08:26 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2310.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 08:26 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2310.codfw.wmnet
  • 08:25 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
  • 08:25 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2310.codfw.wmnet
  • 08:25 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2309.codfw.wmnet
  • 08:25 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
  • 08:25 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
  • 08:24 oblivian@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/aux-k8s-services/jaeger: sync
  • 08:24 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply
  • 08:24 oblivian@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/aux-k8s-services/jaeger: sync
  • 08:24 oblivian@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/jaeger: sync
  • 08:24 oblivian@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/jaeger: sync
  • 08:24 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2309.codfw.wmnet
  • 08:24 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2309.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 08:18 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2309.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 08:18 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2309.codfw.wmnet
  • 08:17 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2309.codfw.wmnet
  • 08:17 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2308.codfw.wmnet
  • 08:17 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.enable_virtualization (exit_code=0) rolling provision on D{wikikube-worker[1303-1330].eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad)
  • 08:17 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1329.eqiad.wmnet
  • 08:16 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1329.eqiad.wmnet
  • 08:16 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2308.codfw.wmnet
  • 08:16 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2308.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 08:15 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1329.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 08:13 hashar@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.47.0-wmf.24 refs T438220
  • 08:11 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2308.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 08:11 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2308.codfw.wmnet
  • 08:09 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2308.codfw.wmnet
  • 08:09 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2307.codfw.wmnet
  • 08:09 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2307.codfw.wmnet
  • 08:09 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1329.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 08:08 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2307.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 08:08 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1329.eqiad.wmnet
  • 08:08 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1329.eqiad.wmnet
  • 08:08 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1328.eqiad.wmnet
  • 08:07 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1328.eqiad.wmnet
  • 08:06 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1328.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 08:03 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2307.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 08:03 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2307.codfw.wmnet
  • 08:02 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2307.codfw.wmnet
  • 08:02 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2306.codfw.wmnet
  • 08:02 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2306.codfw.wmnet
  • 08:01 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2306.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 08:01 logmsgbot: mszwarc Deployed security patch for T440126
  • 08:00 isaranto@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
  • 07:59 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1328.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 07:59 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1328.eqiad.wmnet
  • 07:58 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1328.eqiad.wmnet
  • 07:58 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1327.eqiad.wmnet
  • 07:58 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1327.eqiad.wmnet
  • 07:58 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1327.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 07:56 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2306.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 07:56 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2306.codfw.wmnet
  • 07:55 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2306.codfw.wmnet
  • 07:55 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2305.codfw.wmnet
  • 07:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2305.codfw.wmnet
  • 07:53 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2305.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 07:52 logmsgbot: mszwarc Deployed security patch for T440126
  • 07:48 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2305.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 07:48 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2305.codfw.wmnet
  • 07:48 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1327.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 07:47 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1327.eqiad.wmnet
  • 07:47 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2305.codfw.wmnet
  • 07:47 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2304.codfw.wmnet
  • 07:47 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1327.eqiad.wmnet
  • 07:47 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1326.eqiad.wmnet
  • 07:46 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2304.codfw.wmnet
  • 07:46 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2304.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 07:46 moritzm: installing chromium security updates
  • 07:45 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1326.eqiad.wmnet
  • 07:45 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1326.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 07:40 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2304.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 07:40 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2304.codfw.wmnet
  • 07:40 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2304.codfw.wmnet
  • 07:39 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2303.codfw.wmnet
  • 07:38 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2303.codfw.wmnet
  • 07:38 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2303.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 07:35 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1326.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 07:35 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1326.eqiad.wmnet
  • 07:35 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1326.eqiad.wmnet
  • 07:34 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1325.eqiad.wmnet
  • 07:33 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1325.eqiad.wmnet
  • 07:33 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2303.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 07:32 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1325.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 07:32 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2303.codfw.wmnet
  • 07:32 tgr_: UTC morning deployments done
  • 07:32 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2303.codfw.wmnet
  • 07:31 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2302.codfw.wmnet
  • 07:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2302.codfw.wmnet
  • 07:30 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2302.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 07:30 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
  • 07:30 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
  • 07:30 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
  • 07:29 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: sync
  • 07:25 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2302.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 07:25 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2302.codfw.wmnet
  • 07:24 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2302.codfw.wmnet
  • 07:24 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2301.codfw.wmnet
  • 07:23 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2301.codfw.wmnet
  • 07:23 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2301.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 07:23 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1325.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 07:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1325.eqiad.wmnet
  • 07:22 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1325.eqiad.wmnet
  • 07:21 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1324.eqiad.wmnet
  • 07:20 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1324.eqiad.wmnet
  • 07:20 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1324.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 07:19 tgr@deploy1003: Finished scap sync-world: Backport for Enable $wgOAuth2ClientCredentialsOwnerAuthCutover (T440873) (duration: 13m 29s)
  • 07:17 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2301.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 07:17 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2292.codfw.wmnet
  • 07:17 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2292.codfw.wmnet
  • 07:17 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2301.codfw.wmnet
  • 07:17 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for wikikube-worker2292.codfw.wmnet
  • 07:17 oblivian@cumin1004: START - Cookbook sre.hosts.remove-downtime for wikikube-worker2292.codfw.wmnet
  • 07:16 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2301.codfw.wmnet
  • 07:16 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2300.codfw.wmnet
  • 07:15 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2300.codfw.wmnet
  • 07:15 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2300.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 07:13 tgr@deploy1003: tgr, coderanimeshpathak04092007: Continuing with deployment
  • 07:10 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2300.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 07:10 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2300.codfw.wmnet
  • 07:10 tgr@deploy1003: tgr, coderanimeshpathak04092007: Backport for Enable $wgOAuth2ClientCredentialsOwnerAuthCutover (T440873) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 07:09 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1324.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 07:09 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1324.eqiad.wmnet
  • 07:09 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2300.codfw.wmnet
  • 07:09 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2299.codfw.wmnet
  • 07:09 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1324.eqiad.wmnet
  • 07:08 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1323.eqiad.wmnet
  • 07:07 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1323.eqiad.wmnet
  • 07:06 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2299.codfw.wmnet
  • 07:06 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1323.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 07:05 tgr@deploy1003: Started scap sync-world: Backport for Enable $wgOAuth2ClientCredentialsOwnerAuthCutover (T440873)
  • 06:52 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2299.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 06:52 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1323.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 06:51 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1323.eqiad.wmnet
  • 06:51 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1323.eqiad.wmnet
  • 06:50 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1322.eqiad.wmnet
  • 06:49 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1322.eqiad.wmnet
  • 06:49 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1322.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 06:47 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2299.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 06:47 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2299.codfw.wmnet
  • 06:46 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2299.codfw.wmnet
  • 06:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2298.codfw.wmnet
  • 06:45 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2298.codfw.wmnet
  • 06:44 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2298.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 06:44 moritzm: installing curl security updats
  • 06:39 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2298.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 06:39 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2298.codfw.wmnet
  • 06:38 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2298.codfw.wmnet
  • 06:38 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2297.codfw.wmnet
  • 06:36 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2297.codfw.wmnet
  • 06:36 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1322.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 06:36 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2297.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 06:36 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1322.eqiad.wmnet
  • 06:35 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1322.eqiad.wmnet
  • 06:35 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1321.eqiad.wmnet
  • 06:33 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1321.eqiad.wmnet
  • 06:32 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1321.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 06:31 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2297.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 06:30 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2297.codfw.wmnet
  • 06:30 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2297.codfw.wmnet
  • 06:30 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2296.codfw.wmnet
  • 06:28 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2296.codfw.wmnet
  • 06:28 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2296.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 06:23 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2296.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 06:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2296.codfw.wmnet
  • 06:22 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1321.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 06:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1321.eqiad.wmnet
  • 06:22 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2296.codfw.wmnet
  • 06:21 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2295.codfw.wmnet
  • 06:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1321.eqiad.wmnet
  • 06:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2295.codfw.wmnet
  • 06:21 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1320.eqiad.wmnet
  • 06:21 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2295.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 06:19 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1320.eqiad.wmnet
  • 06:19 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1320.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 06:15 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2295.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 06:15 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2295.codfw.wmnet
  • 06:15 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2295.codfw.wmnet
  • 06:14 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2294.codfw.wmnet
  • 06:13 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2294.codfw.wmnet
  • 06:13 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2294.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 06:09 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1320.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 06:08 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1320.eqiad.wmnet
  • 06:08 oblivian@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 14 days, 0:00:00 on wikikube-worker2292.codfw.wmnet with reason: Hw failure
  • 06:08 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1320.eqiad.wmnet
  • 06:08 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2294.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 06:08 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1319.eqiad.wmnet
  • 06:07 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2294.codfw.wmnet
  • 06:07 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2294.codfw.wmnet
  • 06:07 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2293.codfw.wmnet
  • 06:06 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1319.eqiad.wmnet
  • 06:05 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1319.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 06:05 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2293.codfw.wmnet
  • 06:05 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2293.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 06:00 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2293.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 05:59 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2293.codfw.wmnet
  • 05:59 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2293.codfw.wmnet
  • 05:59 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D{wikikube-worker[2293-2320].codfw.wmnet} and (A:wikikube-master-codfw or A:wikikube-worker-codfw)
  • 05:56 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1319.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 05:55 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1319.eqiad.wmnet
  • 05:55 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1319.eqiad.wmnet
  • 05:55 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1318.eqiad.wmnet
  • 05:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1318.eqiad.wmnet
  • 05:53 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1318.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 05:52 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.enable_virtualization (exit_code=1) rolling provision on D{wikikube-worker[2291-2320].codfw.wmnet} and (A:wikikube-master-codfw or A:wikikube-worker-codfw)
  • 05:43 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1318.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 05:43 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1318.eqiad.wmnet
  • 05:42 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1318.eqiad.wmnet
  • 05:42 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1317.eqiad.wmnet
  • 05:41 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1300.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 05:40 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1317.eqiad.wmnet
  • 05:39 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1317.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 05:30 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1317.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 05:30 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1300.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 05:30 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1317.eqiad.wmnet
  • 05:29 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1317.eqiad.wmnet
  • 05:29 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1316.eqiad.wmnet
  • 05:29 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1316.eqiad.wmnet
  • 05:28 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1316.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 05:19 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1316.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 05:18 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1316.eqiad.wmnet
  • 05:18 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1316.eqiad.wmnet
  • 05:17 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1315.eqiad.wmnet
  • 05:15 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2292.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 05:15 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1315.eqiad.wmnet
  • 05:15 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1315.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 05:14 oblivian@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-worker1300.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 05:13 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1300.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 05:13 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host wikikube-worker1300.eqiad.wmnet
  • 05:10 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1300.eqiad.wmnet
  • 05:10 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2292.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 05:10 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2292.codfw.wmnet
  • 05:09 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1300.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 05:09 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1300.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 05:09 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2292.codfw.wmnet
  • 05:09 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2291.codfw.wmnet
  • 05:09 oblivian@cumin1004: END (ERROR) - Cookbook sre.hosts.provision (exit_code=97) for host wikikube-worker1300.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 05:09 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2291.codfw.wmnet
  • 05:08 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2291.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 05:08 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1300.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 05:05 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1315.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 05:05 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1315.eqiad.wmnet
  • 05:04 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1315.eqiad.wmnet
  • 05:04 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1314.eqiad.wmnet
  • 05:04 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1314.eqiad.wmnet
  • 05:03 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1314.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 05:03 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2291.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 05:03 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2291.codfw.wmnet
  • 05:02 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.sanitize-wiki (exit_code=0) Managing sanitization for wikis mlwikivoyage in section s5
  • 05:02 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2291.codfw.wmnet
  • 05:02 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D{wikikube-worker[2291-2320].codfw.wmnet} and (A:wikikube-master-codfw or A:wikikube-worker-codfw)
  • 04:55 marostegui@cumin1004: START - Cookbook sre.mysql.sanitize-wiki Managing sanitization for wikis mlwikivoyage in section s5
  • 04:54 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1314.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 04:53 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1314.eqiad.wmnet
  • 04:52 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1314.eqiad.wmnet
  • 04:52 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1313.eqiad.wmnet
  • 04:52 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1313.eqiad.wmnet
  • 04:52 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1313.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 04:43 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1313.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 04:43 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1313.eqiad.wmnet
  • 04:42 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1313.eqiad.wmnet
  • 04:42 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1312.eqiad.wmnet
  • 04:40 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1312.eqiad.wmnet
  • 04:39 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1312.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 04:30 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1312.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 04:29 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1312.eqiad.wmnet
  • 04:29 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1312.eqiad.wmnet
  • 04:29 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1311.eqiad.wmnet
  • 04:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1311.eqiad.wmnet
  • 04:27 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1311.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 04:17 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1311.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 04:16 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1311.eqiad.wmnet
  • 04:16 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1311.eqiad.wmnet
  • 04:16 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1310.eqiad.wmnet
  • 04:16 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1310.eqiad.wmnet
  • 04:15 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1310.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 04:12 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2195.codfw.wmnet
  • 04:12 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2195.codfw.wmnet
  • 04:12 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2192.codfw.wmnet
  • 04:12 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2192.codfw.wmnet
  • 04:11 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for wikikube-worker2195.codfw.wmnet
  • 04:11 oblivian@cumin1004: START - Cookbook sre.hosts.remove-downtime for wikikube-worker2195.codfw.wmnet
  • 04:11 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for wikikube-worker2192.codfw.wmnet
  • 04:11 oblivian@cumin1004: START - Cookbook sre.hosts.remove-downtime for wikikube-worker2192.codfw.wmnet
  • 02:54 tstarling@deploy1003: Finished scap sync-world: Backport for Fix Script error: You must specify a function to call. erroring when a function returns nil before other values (T439809) (1.47.0-wmf.23, 1.47.0-wmf.24) (duration: 10m 51s)
  • 02:49 tstarling@deploy1003: tstarling: Continuing with deployment
  • 02:48 tstarling@deploy1003: tstarling: Backport for Fix Script error: You must specify a function to call. erroring when a function returns nil before other values (T439809) (1.47.0-wmf.23, 1.47.0-wmf.24) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 02:43 tstarling@deploy1003: Started scap sync-world: Backport for Fix Script error: You must specify a function to call. erroring when a function returns nil before other values (T439809) (1.47.0-wmf.23, 1.47.0-wmf.24)
  • 02:39 andrewbogott: reverted live-hacking nftables max connection throttle from 32 to 16 on gitlab1004; didn't help. T441247
  • 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 23s)
  • 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image
  • 01:51 andrewbogott: live-hacking nftables max connection throttle from 32 to 16 on gitlab1004 -- T441247
  • 00:00 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
  • 00:00 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: sync

2026-10-14

  • 23:30 RoanKattouw: Deployed security patches for T441041, T441090, T441030, T437940
  • 23:01 zabe@deploy1003: mwscript-k8s job started: foreachwikiindblist wikidataclient extensions/Wikibase/lib/maintenance/populateSitesTable.php --force-protocol https # T441177
  • 22:53 zabe@deploy1003: Finished scap sync-world: Backport for Activate mlwikivoyage (T441169) (duration: 26m 06s)
  • 22:48 zabe@deploy1003: zabe: Continuing with deployment
  • 22:32 zabe@deploy1003: zabe: Backport for Activate mlwikivoyage (T441169) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 22:27 zabe@deploy1003: Started scap sync-world: Backport for Activate mlwikivoyage (T441169)
  • 22:20 zabe: create Wikivoyage Malayalam # T441169
  • 22:05 zabe@deploy1003: Finished scap sync-world: Backport for Prepare mlwikivoyage (T441169) (duration: 10m 44s)
  • 22:00 zabe@deploy1003: zabe: Continuing with deployment
  • 21:59 zabe@deploy1003: zabe: Backport for Prepare mlwikivoyage (T441169) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 21:55 zabe@deploy1003: Started scap sync-world: Backport for Prepare mlwikivoyage (T441169)
  • 21:30 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
  • 21:30 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
  • 21:29 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
  • 21:29 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
  • 21:27 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
  • 21:26 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
  • 21:21 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
  • 21:21 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
  • 21:20 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
  • 21:20 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
  • 21:19 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
  • 21:18 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
  • 21:14 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
  • 21:14 jforrester@deploy1003: Finished scap sync-world: Backport for abstractwiki: Add four more year articles to generate (duration: 10m 10s)
  • 21:14 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
  • 21:13 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1310.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 21:13 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
  • 21:13 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1310.eqiad.wmnet
  • 21:13 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1310.eqiad.wmnet
  • 21:12 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
  • 21:12 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1309.eqiad.wmnet
  • 21:11 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
  • 21:10 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1309.eqiad.wmnet
  • 21:10 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
  • 21:10 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1309.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 21:09 jforrester@deploy1003: jforrester: Continuing with deployment
  • 21:08 jforrester@deploy1003: jforrester: Backport for abstractwiki: Add four more year articles to generate synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 21:04 jforrester@deploy1003: Started scap sync-world: Backport for abstractwiki: Add four more year articles to generate
  • 20:57 akhatun@deploy1003: helmfile [staging] DONE helmfile.d/services/lac-lambdas: apply
  • 20:57 akhatun@deploy1003: helmfile [staging] START helmfile.d/services/lac-lambdas: apply
  • 20:56 akhatun@deploy1003: helmfile [staging] DONE helmfile.d/services/lac-lambdas: apply
  • 20:56 akhatun@deploy1003: helmfile [staging] START helmfile.d/services/lac-lambdas: apply
  • 20:55 akhatun@deploy1003: helmfile [staging] DONE helmfile.d/services/lac-lambdas: apply
  • 20:55 akhatun@deploy1003: helmfile [staging] START helmfile.d/services/lac-lambdas: apply
  • 20:45 wmde-fisch@deploy1003: Finished scap sync-world: Backport for Check for undefined when using mainListIndex (T441193) (1.47.0-wmf.24) (duration: 09m 59s)
  • 20:39 wmde-fisch@deploy1003: wmde-fisch: Continuing with deployment
  • 20:39 wmde-fisch@deploy1003: wmde-fisch: Backport for Check for undefined when using mainListIndex (T441193) (1.47.0-wmf.24) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 20:35 wmde-fisch@deploy1003: Started scap sync-world: Backport for Check for undefined when using mainListIndex (T441193) (1.47.0-wmf.24)
  • 20:33 wmde-fisch@deploy1003: Finished scap sync-world: Backport for Check for undefined when using mainListIndex (T441193) (1.47.0-wmf.23) (duration: 10m 10s)
  • 20:33 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=cp2046.*
  • 20:33 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp2046.codfw.wmnet
  • 20:32 cdobbins@cumin1004: START - Cookbook sre.hosts.remove-downtime for cp2046.codfw.wmnet
  • 20:31 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp2046.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 20:27 wmde-fisch@deploy1003: wmde-fisch: Continuing with deployment
  • 20:27 wmde-fisch@deploy1003: wmde-fisch: Backport for Check for undefined when using mainListIndex (T441193) (1.47.0-wmf.23) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 20:23 wmde-fisch@deploy1003: Started scap sync-world: Backport for Check for undefined when using mainListIndex (T441193) (1.47.0-wmf.23)
  • 20:22 kemayo@deploy1003: Finished scap sync-world: Backport for VisualEditor: Precompute edit suggestion counts on 25% of enwiki (T433024), ExtensionDistributor: Mark 1.47 as beta (T441077), InitialiseSettings: Enable ModeratorToolkit on four group2 wikis (T437617), InitialiseSettings: Enable ModeratorToolkit on enwiki (T437617) (duration: 14m 26s)
  • 20:19 cdobbins@cumin1004: START - Cookbook sre.hosts.provision for host cp2046.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 20:17 cdobbins@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts cp2046.codfw.wmnet
  • 20:17 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cp2046.codfw.wmnet
  • 20:16 kemayo@deploy1003: ariel, kemayo, dhardy: Continuing with deployment
  • 20:12 kemayo@deploy1003: ariel, kemayo, dhardy: Backport for VisualEditor: Precompute edit suggestion counts on 25% of enwiki (T433024), ExtensionDistributor: Mark 1.47 as beta (T441077), InitialiseSettings: Enable ModeratorToolkit on four group2 wikis (T437617), InitialiseSettings: Enable ModeratorToolkit on enwiki (T437617) synced to the testserve
  • 20:08 kemayo@deploy1003: Started scap sync-world: Backport for VisualEditor: Precompute edit suggestion counts on 25% of enwiki (T433024), ExtensionDistributor: Mark 1.47 as beta (T441077), InitialiseSettings: Enable ModeratorToolkit on four group2 wikis (T437617), InitialiseSettings: Enable ModeratorToolkit on enwiki (T437617)
  • 19:56 cdobbins@cumin1004: START - Cookbook sre.hosts.reboot-single for host cp2046.codfw.wmnet
  • 19:52 cdobbins@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cp2046.codfw.wmnet
  • 19:52 cdobbins@cumin1004: conftool action : set/pooled=no; selector: name=cp2046.*
  • 19:40 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=cp2045.*
  • 19:40 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp2045.codfw.wmnet
  • 19:40 cdobbins@cumin1004: START - Cookbook sre.hosts.remove-downtime for cp2045.codfw.wmnet
  • 19:37 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp2045.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 19:22 cdobbins@cumin1004: START - Cookbook sre.hosts.provision for host cp2045.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 19:21 kemayo@deploy1003: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply
  • 19:20 kemayo@deploy1003: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply
  • 19:20 kemayo@deploy1003: helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply
  • 19:19 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cp2045.codfw.wmnet
  • 19:19 kemayo@deploy1003: helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply
  • 19:17 kemayo@deploy1003: helmfile [staging] DONE helmfile.d/services/changeprop-jobqueue: apply
  • 19:16 kemayo@deploy1003: helmfile [staging] START helmfile.d/services/changeprop-jobqueue: apply
  • 19:15 kemayo@deploy1003: helmfile [codfw] DONE helmfile.d/services/editcheck-headless: apply
  • 19:14 kemayo@deploy1003: helmfile [codfw] START helmfile.d/services/editcheck-headless: apply
  • 19:14 kemayo@deploy1003: helmfile [eqiad] DONE helmfile.d/services/editcheck-headless: apply
  • 19:14 kemayo@deploy1003: helmfile [eqiad] START helmfile.d/services/editcheck-headless: apply
  • 19:12 cdobbins@cumin1004: START - Cookbook sre.hosts.reboot-single for host cp2045.codfw.wmnet
  • 19:11 cdobbins@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts cp2045.codfw.wmnet
  • 19:11 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cp2045.codfw.wmnet
  • 18:55 akhatun@deploy1003: helmfile [staging] DONE helmfile.d/services/lac-lambdas: apply
  • 18:55 akhatun@deploy1003: helmfile [staging] START helmfile.d/services/lac-lambdas: apply
  • 18:55 akhatun@deploy1003: helmfile [staging] DONE helmfile.d/services/lac-lambdas: apply
  • 18:54 akhatun@deploy1003: helmfile [staging] START helmfile.d/services/lac-lambdas: apply
  • 18:50 akhatun@deploy1003: helmfile [staging] DONE helmfile.d/services/lac-lambdas: apply
  • 18:50 akhatun@deploy1003: helmfile [staging] START helmfile.d/services/lac-lambdas: apply
  • 18:50 dzahn@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on codesearch2001.codfw.wmnet with reason: in setup, frontend image pending
  • 18:49 cdobbins@cumin1004: START - Cookbook sre.hosts.reboot-single for host cp2045.codfw.wmnet
  • 18:47 cdobbins@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cp2045.codfw.wmnet
  • 18:47 cdobbins@cumin1004: conftool action : set/pooled=no; selector: name=cp2045.*
  • 18:22 brett@cumin2003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp2044.codfw.wmnet
  • 18:22 brett@cumin2003: START - Cookbook sre.hosts.remove-downtime for cp2044.codfw.wmnet
  • 18:19 brennen@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.24 refs T438220
  • 18:12 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.enable_virtualization (exit_code=0) rolling provision on D{wikikube-worker[2261-2290].codfw.wmnet} and (A:wikikube-master-codfw or A:wikikube-worker-codfw)
  • 18:12 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2290.codfw.wmnet
  • 18:11 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2290.codfw.wmnet
  • 18:11 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2290.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 18:06 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2290.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 18:05 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2290.codfw.wmnet
  • 18:05 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2290.codfw.wmnet
  • 18:04 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2289.codfw.wmnet
  • 18:04 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2289.codfw.wmnet
  • 18:04 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2289.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 17:58 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2289.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 17:58 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2289.codfw.wmnet
  • 17:57 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2289.codfw.wmnet
  • 17:57 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2288.codfw.wmnet
  • 17:56 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2288.codfw.wmnet
  • 17:56 isaranto@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
  • 17:56 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2288.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 17:51 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2288.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 17:51 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2288.codfw.wmnet
  • 17:50 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2288.codfw.wmnet
  • 17:50 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2287.codfw.wmnet
  • 17:49 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2287.codfw.wmnet
  • 17:48 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2287.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 17:43 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2287.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 17:43 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2287.codfw.wmnet
  • 17:42 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2287.codfw.wmnet
  • 17:42 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2286.codfw.wmnet
  • 17:42 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2286.codfw.wmnet
  • 17:41 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2286.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 17:36 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2286.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 17:36 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2286.codfw.wmnet
  • 17:35 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=cp2044.*
  • 17:35 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2286.codfw.wmnet
  • 17:35 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2285.codfw.wmnet
  • 17:35 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2285.codfw.wmnet
  • 17:27 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp2044.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 17:25 sukhe@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on cp2044.codfw.wmnet with reason: host is depooled, rebooting
  • 17:24 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2285.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 17:19 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2285.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 17:18 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2285.codfw.wmnet
  • 17:18 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2285.codfw.wmnet
  • 17:18 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2284.codfw.wmnet
  • 17:15 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2284.codfw.wmnet
  • 17:15 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2284.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 17:11 cdobbins@cumin1004: START - Cookbook sre.hosts.provision for host cp2044.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 17:09 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2284.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 17:09 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2284.codfw.wmnet
  • 17:09 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cp2044.codfw.wmnet
  • 17:09 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2284.codfw.wmnet
  • 17:08 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2283.codfw.wmnet
  • 17:07 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2283.codfw.wmnet
  • 17:07 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2283.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 17:02 cdobbins@cumin1004: START - Cookbook sre.hosts.reboot-single for host cp2044.codfw.wmnet
  • 17:02 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2283.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 17:01 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2283.codfw.wmnet
  • 17:01 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2283.codfw.wmnet
  • 17:01 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2282.codfw.wmnet
  • 16:59 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2282.codfw.wmnet
  • 16:59 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2282.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 16:56 kamila@deploy1003: Finished scap deploy-service shellbox: roll out php 8.5 on shellbox-media (duration: 03m 05s)
  • 16:56 cdobbins@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts cp2044.codfw.wmnet
  • 16:56 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cp2044.codfw.wmnet
  • 16:56 kamila@deploy1003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
  • 16:55 kamila@deploy1003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] codfw Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
  • 16:54 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2282.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 16:53 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2282.codfw.wmnet
  • 16:53 kamila@deploy1003: scap deploy-service shellbox: [STAGING/DEFAULT] staging-eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
  • 16:53 kamila@deploy1003: Started scap deploy-service shellbox: roll out php 8.5 on shellbox-media
  • 16:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2282.codfw.wmnet
  • 16:53 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2281.codfw.wmnet
  • 16:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2281.codfw.wmnet
  • 16:52 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2281.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 16:47 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2281.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 16:47 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2281.codfw.wmnet
  • 16:46 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2281.codfw.wmnet
  • 16:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2280.codfw.wmnet
  • 16:46 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2280.codfw.wmnet
  • 16:46 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2280.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 16:41 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2280.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 16:40 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2280.codfw.wmnet
  • 16:40 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2280.codfw.wmnet
  • 16:39 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2279.codfw.wmnet
  • 16:38 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2279.codfw.wmnet
  • 16:38 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2279.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 16:34 cdobbins@cumin1004: START - Cookbook sre.hosts.reboot-single for host cp2044.codfw.wmnet
  • 16:33 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2279.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 16:32 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2279.codfw.wmnet
  • 16:32 cdobbins@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cp2044.codfw.wmnet
  • 16:32 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2279.codfw.wmnet
  • 16:31 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2278.codfw.wmnet
  • 16:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2278.codfw.wmnet
  • 16:31 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2278.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 16:26 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2278.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 16:26 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2278.codfw.wmnet
  • 16:25 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2278.codfw.wmnet
  • 16:25 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2277.codfw.wmnet
  • 16:24 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1309.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 16:24 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1309.eqiad.wmnet
  • 16:24 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2277.codfw.wmnet
  • 16:23 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2277.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 16:23 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1309.eqiad.wmnet
  • 16:23 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1308.eqiad.wmnet
  • 16:22 cdobbins@cumin1004: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts cp2044.codfw.wmnet
  • 16:22 cdobbins@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cp2044.codfw.wmnet
  • 16:22 cdobbins@cumin1004: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts cp2044.codfw.wmnet
  • 16:22 cdobbins@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cp2044.codfw.wmnet
  • 16:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1308.eqiad.wmnet
  • 16:20 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1308.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 16:20 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen: apply
  • 16:20 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen: apply
  • 16:18 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2277.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 16:18 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2277.codfw.wmnet
  • 16:17 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2277.codfw.wmnet
  • 16:17 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2276.codfw.wmnet
  • 16:16 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2276.codfw.wmnet
  • 16:16 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2276.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 16:11 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1308.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 16:10 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2276.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 16:10 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1308.eqiad.wmnet
  • 16:10 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2276.codfw.wmnet
  • 16:10 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1308.eqiad.wmnet
  • 16:10 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2276.codfw.wmnet
  • 16:09 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1307.eqiad.wmnet
  • 16:09 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2275.codfw.wmnet
  • 16:09 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1307.eqiad.wmnet
  • 16:08 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1307.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 16:08 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2275.codfw.wmnet
  • 16:08 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2275.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 16:05 ladsgroup@dns1004: END - running authdns-update
  • 16:03 ladsgroup@dns1004: START - running authdns-update
  • 16:02 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2275.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 16:02 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2275.codfw.wmnet
  • 16:02 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2275.codfw.wmnet
  • 16:01 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2274.codfw.wmnet
  • 16:01 cdobbins@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts cp2044.codfw.wmnet
  • 16:00 cdobbins@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cp2044.codfw.wmnet
  • 16:00 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2274.codfw.wmnet
  • 16:00 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2274.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 15:57 cdobbins@cumin1004: conftool action : set/pooled=no; selector: name=cp2044.*
  • 15:55 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2274.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 15:54 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2274.codfw.wmnet
  • 15:54 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2274.codfw.wmnet
  • 15:53 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2273.codfw.wmnet
  • 15:52 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2273.codfw.wmnet
  • 15:52 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2273.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 15:47 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2273.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 15:47 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2273.codfw.wmnet
  • 15:46 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2273.codfw.wmnet
  • 15:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2272.codfw.wmnet
  • 15:45 kamila@deploy1003: Finished scap deploy-service shellbox: roll out PHP 8.5 on shellbox-constraints (duration: 06m 32s)
  • 15:45 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2272.codfw.wmnet
  • 15:45 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2272.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 15:41 kamila@deploy1003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
  • 15:40 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2272.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 15:40 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2272.codfw.wmnet
  • 15:39 kamila@deploy1003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] codfw Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
  • 15:39 akhatun@deploy1003: helmfile [staging] DONE helmfile.d/services/lac-lambdas: apply
  • 15:39 akhatun@deploy1003: helmfile [staging] START helmfile.d/services/lac-lambdas: apply
  • 15:39 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2272.codfw.wmnet
  • 15:39 kamila@deploy1003: scap deploy-service shellbox: [STAGING/DEFAULT] staging-eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
  • 15:39 kamila@deploy1003: Started scap deploy-service shellbox: roll out PHP 8.5 on shellbox-constraints
  • 15:39 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2271.codfw.wmnet
  • 15:38 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2271.codfw.wmnet
  • 15:38 elukey@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on pki1002.eqiad.wmnet with reason: T434268
  • 15:38 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2271.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 15:33 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2271.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 15:32 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2271.codfw.wmnet
  • 15:32 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2271.codfw.wmnet
  • 15:32 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2270.codfw.wmnet
  • 15:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2270.codfw.wmnet
  • 15:31 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2270.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 15:26 elukey@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts sretest2003.codfw.wmnet
  • 15:26 elukey@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts sretest2003.codfw.wmnet
  • 15:25 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2270.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 15:25 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2270.codfw.wmnet
  • 15:24 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2270.codfw.wmnet
  • 15:24 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2269.codfw.wmnet
  • 15:24 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1307.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 15:24 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1307.eqiad.wmnet
  • 15:23 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1307.eqiad.wmnet
  • 15:23 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1306.eqiad.wmnet
  • 15:23 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1306.eqiad.wmnet
  • 15:23 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2269.codfw.wmnet
  • 15:23 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2269.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 15:23 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1306.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 15:21 aikochou@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
  • 15:18 papaul: maintenance on mr1-eqiad complete
  • 15:17 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2269.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 15:17 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2269.codfw.wmnet
  • 15:17 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2269.codfw.wmnet
  • 15:16 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2268.codfw.wmnet
  • 15:15 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2268.codfw.wmnet
  • 15:15 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2268.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 15:14 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1306.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 15:13 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1306.eqiad.wmnet
  • 15:13 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1306.eqiad.wmnet
  • 15:13 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1305.eqiad.wmnet
  • 15:13 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1305.eqiad.wmnet
  • 15:12 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1305.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 15:10 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2268.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 15:09 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2268.codfw.wmnet
  • 15:09 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2268.codfw.wmnet
  • 15:09 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2267.codfw.wmnet
  • 15:08 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2267.codfw.wmnet
  • 15:08 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2267.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 15:04 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1305.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 15:04 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1305.eqiad.wmnet
  • 15:03 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1305.eqiad.wmnet
  • 15:03 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1304.eqiad.wmnet
  • 15:02 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2267.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 15:02 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2267.codfw.wmnet
  • 15:02 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2267.codfw.wmnet
  • 15:01 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2266.codfw.wmnet
  • 15:01 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2266.codfw.wmnet
  • 15:00 elukey: powercycle pki1002, unreachable via ssh, continuous stack dumps on the tty via mgmg console
  • 15:00 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1304.eqiad.wmnet
  • 15:00 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2266.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 15:00 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1304.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 14:55 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2266.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 14:55 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2266.codfw.wmnet
  • 14:54 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2266.codfw.wmnet
  • 14:54 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2265.codfw.wmnet
  • 14:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2265.codfw.wmnet
  • 14:53 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2265.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 14:52 dancy@deploy1003: Installation of scap version "4.298.0" completed for 158 hosts
  • 14:50 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1304.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 14:50 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1304.eqiad.wmnet
  • 14:50 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply
  • 14:50 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp2043.codfw.wmnet
  • 14:50 cdobbins@cumin1004: START - Cookbook sre.hosts.remove-downtime for cp2043.codfw.wmnet
  • 14:50 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply
  • 14:49 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1304.eqiad.wmnet
  • 14:49 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1303.eqiad.wmnet
  • 14:49 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1303.eqiad.wmnet
  • 14:48 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1303.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 14:48 dancy@deploy1003: Installing scap version "4.298.0" for 158 host(s)
  • 14:48 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2265.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 14:47 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2265.codfw.wmnet
  • 14:46 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2265.codfw.wmnet
  • 14:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2264.codfw.wmnet
  • 14:46 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2264.codfw.wmnet
  • 14:46 oblivian@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-worker2264.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 14:46 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2264.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 14:39 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1303.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 14:38 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1303.eqiad.wmnet
  • 14:38 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1303.eqiad.wmnet
  • 14:38 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D{wikikube-worker[1303-1330].eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad)
  • 14:33 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.enable_virtualization (exit_code=1) rolling provision on D{wikikube-worker[1300-1330].eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad)
  • 14:33 oblivian@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-worker1302.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 14:30 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1302.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 14:28 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=cp2043.*
  • 14:15 jforrester@deploy1003: Finished scap sync-world: Backport for dagwiki & igwiki: Enable Abstract Wikimedia client mode, wikifunctions: Drop CodeMirror enablement, defailt on (T400015) (duration: 10m 14s)
  • 14:09 jforrester@deploy1003: jforrester: Continuing with deployment
  • 14:09 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply
  • 14:09 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply
  • 14:09 jforrester@deploy1003: jforrester: Backport for dagwiki & igwiki: Enable Abstract Wikimedia client mode, wikifunctions: Drop CodeMirror enablement, defailt on (T400015) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 14:04 jforrester@deploy1003: Started scap sync-world: Backport for dagwiki & igwiki: Enable Abstract Wikimedia client mode, wikifunctions: Drop CodeMirror enablement, defailt on (T400015)
  • 14:01 oblivian@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-worker2264.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 14:01 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2264.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 14:00 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2264.codfw.wmnet
  • 14:00 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2264.codfw.wmnet
  • 14:00 oblivian@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-worker1302.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 14:00 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2263.codfw.wmnet
  • 13:59 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2263.codfw.wmnet
  • 13:58 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2263.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 13:53 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2263.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 13:53 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2263.codfw.wmnet
  • 13:52 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2263.codfw.wmnet
  • 13:52 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2262.codfw.wmnet
  • 13:51 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 8 hosts with reason: Junos upgrade
  • 13:51 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2262.codfw.wmnet
  • 13:51 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2262.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 13:50 pt1979@cumin1004: DONE (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on 8 hosts with reason: Junos upgrade
  • 13:50 papaul: start maintenance on mr1-eqiad
  • 13:48 kamila@deploy1003: Finished scap deploy-service shellbox: pilot PHP 8.5 on shellbox-media (duration: 02m 29s)
  • 13:47 kamila@deploy1003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
  • 13:47 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1302.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 13:47 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1302.eqiad.wmnet
  • 13:46 kamila@deploy1003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] codfw Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
  • 13:45 kamila@deploy1003: scap deploy-service shellbox: [STAGING/DEFAULT] staging-eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
  • 13:45 kamila@deploy1003: Started scap deploy-service shellbox: pilot PHP 8.5 on shellbox-media
  • 13:45 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2262.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 13:45 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2262.codfw.wmnet
  • 13:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2262.codfw.wmnet
  • 13:44 krinkle@deploy1003: Finished scap sync-world: Backport for WikiMap: Let $wgConf take precedence over "current wiki" (T440997 T428296) (1.47.0-wmf.24) (duration: 13m 08s)
  • 13:44 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2261.codfw.wmnet
  • 13:43 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2261.codfw.wmnet
  • 13:43 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2261.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 13:43 daniel@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply
  • 13:42 daniel@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply
  • 13:42 oblivian@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 14 days, 0:00:00 on wikikube-worker1298.eqiad.wmnet with reason: Hw failure
  • 13:42 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1302.eqiad.wmnet
  • 13:42 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1301.eqiad.wmnet
  • 13:41 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1301.eqiad.wmnet
  • 13:41 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1301.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 13:40 oblivian@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 14 days, 0:00:00 on wikikube-worker2192.codfw.wmnet with reason: Hw failure
  • 13:39 krinkle@deploy1003: krinkle: Continuing with deployment
  • 13:39 oblivian@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 14 days, 0:00:00 on wikikube-worker2195.codfw.wmnet with reason: Hw failure
  • 13:38 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2261.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 13:38 cgoubert@cumin2003: END (PASS) - Cookbook sre.memcached.roll-reboot-restart (exit_code=0) rolling restart_daemons on A:memcached-codfw
  • 13:38 javiermonton@deploy1003: Finished deploy [analytics/refinery@5faed81]: Regular analytics weekly train [analytics/refinery@5faed811] (duration: 04m 31s)
  • 13:38 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2261.codfw.wmnet
  • 13:37 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2261.codfw.wmnet
  • 13:37 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D{wikikube-worker[2261-2290].codfw.wmnet} and (A:wikikube-master-codfw or A:wikikube-worker-codfw)
  • 13:35 krinkle@deploy1003: krinkle: Backport for WikiMap: Let $wgConf take precedence over "current wiki" (T440997 T428296) (1.47.0-wmf.24) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 13:35 daniel@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply
  • 13:35 daniel@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply
  • 13:33 javiermonton@deploy1003: Started deploy [analytics/refinery@5faed81]: Regular analytics weekly train [analytics/refinery@5faed811]
  • 13:32 javiermonton@deploy1003: Finished deploy [analytics/refinery@5faed81] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@5faed811] (duration: 00m 07s)
  • 13:32 javiermonton@deploy1003: Started deploy [analytics/refinery@5faed81] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@5faed811]
  • 13:32 javiermonton@deploy1003: Finished deploy [analytics/refinery@5faed81] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@5faed811] (duration: 00m 45s)
  • 13:31 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1301.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 13:31 krinkle@deploy1003: Started scap sync-world: Backport for WikiMap: Let $wgConf take precedence over "current wiki" (T440997 T428296) (1.47.0-wmf.24)
  • 13:31 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1301.eqiad.wmnet
  • 13:31 javiermonton@deploy1003: Started deploy [analytics/refinery@5faed81] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@5faed811]
  • 13:30 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1301.eqiad.wmnet
  • 13:30 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D{wikikube-worker[1300-1330].eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad)
  • 13:29 JavierMonton: Deploying Refinery at 5faed811 for changes 1352228, 1344135, 1354754
  • 13:27 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.enable_virtualization (exit_code=0) rolling provision on D{wikikube-worker[1299-1300].eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad)
  • 13:27 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1300.eqiad.wmnet
  • 13:27 daniel@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply
  • 13:26 daniel@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply
  • 13:25 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1300.eqiad.wmnet
  • 13:25 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1300.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 13:25 krinkle@deploy1003: Finished scap sync-world: Backport for Enable mobile editing for all Wikidata editors (T437054) (duration: 13m 14s)
  • 13:20 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
  • 13:20 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
  • 13:20 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
  • 13:20 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: sync
  • 13:19 krinkle@deploy1003: krinkle, sadiyamohammed13: Continuing with deployment
  • 13:17 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.enable_virtualization (exit_code=0) rolling provision on D{wikikube-worker[2088,2203,2242-2243,2248-2260].codfw.wmnet} and (A:wikikube-master-codfw or A:wikikube-worker-codfw)
  • 13:17 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2260.codfw.wmnet
  • 13:16 krinkle@deploy1003: krinkle, sadiyamohammed13: Backport for Enable mobile editing for all Wikidata editors (T437054) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 13:15 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2260.codfw.wmnet
  • 13:15 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2260.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 13:15 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1300.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 13:14 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1300.eqiad.wmnet
  • 13:14 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1300.eqiad.wmnet
  • 13:14 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1299.eqiad.wmnet
  • 13:14 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1299.eqiad.wmnet
  • 13:13 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1299.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 13:11 krinkle@deploy1003: Started scap sync-world: Backport for Enable mobile editing for all Wikidata editors (T437054)
  • 13:10 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2260.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 13:10 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2260.codfw.wmnet
  • 13:09 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2260.codfw.wmnet
  • 13:09 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2259.codfw.wmnet
  • 13:08 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2259.codfw.wmnet
  • 13:07 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2259.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 13:07 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
  • 13:03 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1299.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 13:03 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1299.eqiad.wmnet
  • 13:03 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1299.eqiad.wmnet
  • 13:02 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D{wikikube-worker[1299-1300].eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad)
  • 13:02 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2259.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 13:02 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2259.codfw.wmnet
  • 13:01 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2259.codfw.wmnet
  • 13:01 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2258.codfw.wmnet
  • 13:01 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2258.codfw.wmnet
  • 13:01 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2258.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 12:56 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.enable_virtualization (exit_code=1) rolling provision on D{wikikube-worker[1291-1300].eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad)
  • 12:55 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2258.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 12:55 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2258.codfw.wmnet
  • 12:54 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2258.codfw.wmnet
  • 12:54 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2257.codfw.wmnet
  • 12:54 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2257.codfw.wmnet
  • 12:54 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2257.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 12:48 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2257.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 12:48 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2257.codfw.wmnet
  • 12:48 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2257.codfw.wmnet
  • 12:48 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2203.codfw.wmnet
  • 12:48 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2203.codfw.wmnet
  • 12:47 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2203.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 12:46 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2203.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 12:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2203.codfw.wmnet
  • 12:45 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2203.codfw.wmnet
  • 12:45 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2088.codfw.wmnet
  • 12:45 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2088.codfw.wmnet
  • 12:43 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2088.codfw.wmnet
  • 12:42 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2088.codfw.wmnet
  • 12:42 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D{wikikube-worker[2088,2203,2242-2243,2248-2260].codfw.wmnet} and (A:wikikube-master-codfw or A:wikikube-worker-codfw)
  • 12:39 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2257.codfw.wmnet
  • 12:39 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2257.codfw.wmnet
  • 12:36 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.enable_virtualization (exit_code=1) rolling provision on D{wikikube-worker[2088,2203,2242-2243,2248-2260].codfw.wmnet} and (A:wikikube-master-codfw or A:wikikube-worker-codfw)
  • 12:35 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1298.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 12:33 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2257.codfw.wmnet
  • 12:33 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2257.codfw.wmnet
  • 12:32 kamila@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-constraints: apply
  • 12:32 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2256.codfw.wmnet
  • 12:32 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2256.codfw.wmnet
  • 12:32 kamila@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-constraints: apply
  • 12:32 kamila@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-constraints: apply
  • 12:32 kamila@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-constraints: apply
  • 12:29 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2256.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 12:28 kamila@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-constraints: apply
  • 12:28 kamila@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-constraints: apply
  • 12:25 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'.
  • 12:24 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'.
  • 12:23 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2256.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 12:20 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
  • 12:19 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
  • 12:14 cgoubert@cumin2003: START - Cookbook sre.memcached.roll-reboot-restart rolling restart_daemons on A:memcached-codfw
  • 12:04 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1298.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 12:04 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1298.eqiad.wmnet
  • 12:04 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1298.eqiad.wmnet
  • 12:03 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1297.eqiad.wmnet
  • 12:01 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1297.eqiad.wmnet
  • 12:01 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1297.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 11:57 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
  • 11:51 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
  • 11:51 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1297.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 11:50 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1297.eqiad.wmnet
  • 11:50 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1297.eqiad.wmnet
  • 11:50 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1296.eqiad.wmnet
  • 11:49 btullis@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'.
  • 11:47 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1296.eqiad.wmnet
  • 11:47 btullis@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'.
  • 11:47 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1296.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 11:44 klausman@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on ml-serve1015.eqiad.wmnet with reason: Downtime for more BIOS setup rummaging
  • 11:43 isaranto@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
  • 11:37 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1296.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 11:37 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1296.eqiad.wmnet
  • 11:36 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1296.eqiad.wmnet
  • 11:36 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1295.eqiad.wmnet
  • 11:36 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1295.eqiad.wmnet
  • 11:36 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1295.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 11:26 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1295.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 11:26 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1295.eqiad.wmnet
  • 11:25 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1295.eqiad.wmnet
  • 11:25 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1294.eqiad.wmnet
  • 11:25 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1294.eqiad.wmnet
  • 11:25 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1294.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 11:15 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1294.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 11:14 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1294.eqiad.wmnet
  • 11:14 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1294.eqiad.wmnet
  • 11:14 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1293.eqiad.wmnet
  • 11:12 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1293.eqiad.wmnet
  • 11:12 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1293.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 11:02 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1293.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 11:02 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1293.eqiad.wmnet
  • 11:01 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1293.eqiad.wmnet
  • 11:01 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1292.eqiad.wmnet
  • 11:00 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2256.codfw.wmnet
  • 11:00 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2256.codfw.wmnet
  • 10:59 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2255.codfw.wmnet
  • 10:59 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2255.codfw.wmnet
  • 10:59 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2255.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 10:59 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1292.eqiad.wmnet
  • 10:59 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1292.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 10:54 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2255.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 10:52 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2255.codfw.wmnet
  • 10:52 claime: disabling memcached extstore in codfw - T440896
  • 10:51 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2255.codfw.wmnet
  • 10:51 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2254.codfw.wmnet
  • 10:51 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2254.codfw.wmnet
  • 10:51 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2254.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 10:49 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1292.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 10:49 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1292.eqiad.wmnet
  • 10:48 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1292.eqiad.wmnet
  • 10:48 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1291.eqiad.wmnet
  • 10:48 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1291.eqiad.wmnet
  • 10:47 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1291.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 10:45 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2254.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 10:44 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2254.codfw.wmnet
  • 10:44 claime: Repooling eqiad before disabling memcached extstore in codfw - T440896
  • 10:43 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2254.codfw.wmnet
  • 10:43 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2253.codfw.wmnet
  • 10:42 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2253.codfw.wmnet
  • 10:42 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2253.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 10:38 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1291.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 10:37 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1291.eqiad.wmnet
  • 10:37 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2253.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 10:37 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1291.eqiad.wmnet
  • 10:37 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D{wikikube-worker[1291-1300].eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad)
  • 10:31 claime: Giving memcache a little bit to warm up, p50 latency is up 50% accross the board - T440896
  • 10:30 claime: Curves crossed, all read traffic is served by codfw - T440896
  • 10:26 cgoubert@cumin2003: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) depool mw-api-int-ro in eqiad: maintenance
  • 10:25 cgoubert@cumin2003: START - Cookbook sre.discovery.service-route depool mw-api-int-ro in eqiad: maintenance
  • 10:24 cgoubert@cumin2003: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) depool mw-api-ext-ro in eqiad: maintenance
  • 10:23 cgoubert@cumin2003: START - Cookbook sre.discovery.service-route depool mw-api-ext-ro in eqiad: maintenance
  • 10:23 cgoubert@cumin2003: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) depool mw-web-ro in eqiad: maintenance
  • 10:22 cgoubert@cumin2003: START - Cookbook sre.discovery.service-route depool mw-web-ro in eqiad: maintenance
  • 10:19 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2253.codfw.wmnet
  • 10:18 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2253.codfw.wmnet
  • 10:18 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2252.codfw.wmnet
  • 10:18 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2252.codfw.wmnet
  • 10:18 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2252.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 10:13 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2252.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 10:07 hashar@deploy1003: Finished deploy [gerrit/gerrit@0d40a02]: Gerrit to 3.10.9-79-g5500b68896 and update plugins - T437620 (duration: 00m 12s)
  • 10:07 hashar@deploy1003: Started deploy [gerrit/gerrit@0d40a02]: Gerrit to 3.10.9-79-g5500b68896 and update plugins - T437620
  • 10:07 hashar: Upgrading Gerrit on gerrit.wikimedia.org (gerrit1003)
  • 10:06 claime: Starting work on T440896 - Lowering TTLs
  • 09:47 arnaudb@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on gerrit[1003,2003].wikimedia.org with reason: Hashar unblocking mediawiki train
  • 09:47 arnaudb@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:20:00 on gerrit[1003,2003].wikimedia.org with reason: Hashar unblocking mediawiki train
  • 09:42 hashar: Upgrading Gerrit on gerrit-replica.wikimedia.org (gerrit2003)
  • 09:41 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.enable_virtualization (exit_code=0) rolling provision on D{wikikube-worker[1260-1289].eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad)
  • 09:41 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1289.eqiad.wmnet
  • 09:41 hashar@deploy1003: Finished deploy [gerrit/gerrit@0d40a02]: Gerrit to 3.10.9-79-g5500b68896 and update plugins - T437620 (duration: 00m 16s)
  • 09:40 hashar@deploy1003: Started deploy [gerrit/gerrit@0d40a02]: Gerrit to 3.10.9-79-g5500b68896 and update plugins - T437620
  • 09:40 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1289.eqiad.wmnet
  • 09:40 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1289.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 09:36 hashar@deploy1003: Finished deploy [gerrit/gerrit@0d40a02]: Expand Gerrit built-in plugins - T437620 (duration: 00m 13s)
  • 09:35 hashar@deploy1003: Started deploy [gerrit/gerrit@0d40a02]: Expand Gerrit built-in plugins - T437620
  • 09:29 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1289.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 09:29 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1289.eqiad.wmnet
  • 09:28 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1289.eqiad.wmnet
  • 09:28 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1288.eqiad.wmnet
  • 09:28 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1288.eqiad.wmnet
  • 09:28 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1288.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 09:20 hashar@deploy1003: Finished deploy [gerrit/gerrit@0b12955]: Gerrit to 3.10.9-79-g5500b68896 and update plugins - T437620 (duration: 00m 14s)
  • 09:20 hashar@deploy1003: Started deploy [gerrit/gerrit@0b12955]: Gerrit to 3.10.9-79-g5500b68896 and update plugins - T437620
  • 09:18 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1288.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 09:18 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1288.eqiad.wmnet
  • 09:17 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1288.eqiad.wmnet
  • 09:17 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1287.eqiad.wmnet
  • 09:16 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1287.eqiad.wmnet
  • 09:15 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1287.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 09:05 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1287.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 09:05 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1287.eqiad.wmnet
  • 09:04 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1287.eqiad.wmnet
  • 09:04 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1286.eqiad.wmnet
  • 09:02 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1286.eqiad.wmnet
  • 09:02 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1286.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 08:58 arnaudb@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on gerrit[1003,2002-2003].wikimedia.org with reason: Hashar unblocking mediawiki train
  • 08:52 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1286.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 08:52 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1286.eqiad.wmnet
  • 08:51 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1286.eqiad.wmnet
  • 08:51 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1285.eqiad.wmnet
  • 08:49 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1285.eqiad.wmnet
  • 08:49 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1285.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 08:41 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/lac-lambdas: apply
  • 08:40 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/lac-lambdas: apply
  • 08:39 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1285.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 08:38 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1285.eqiad.wmnet
  • 08:37 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1285.eqiad.wmnet
  • 08:37 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1284.eqiad.wmnet
  • 08:37 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1284.eqiad.wmnet
  • 08:37 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1284.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 08:36 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2252.codfw.wmnet
  • 08:35 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2252.codfw.wmnet
  • 08:35 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2251.codfw.wmnet
  • 08:34 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2251.codfw.wmnet
  • 08:34 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2251.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 08:32 arnaudb: gerrit2002: the spare Gerrit is back on ldap-ro.codfw, restarted - T440888
  • 08:31 arnaudb@cumin1004: END (PASS) - Cookbook sre.gerrit.restart-gerrit (exit_code=0) Restarting Gerrit on gerrit2002
  • 08:30 arnaudb@cumin1004: START - Cookbook sre.gerrit.restart-gerrit Restarting Gerrit on gerrit2002
  • 08:28 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2251.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 08:27 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1284.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 08:27 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1284.eqiad.wmnet
  • 08:26 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1284.eqiad.wmnet
  • 08:26 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1283.eqiad.wmnet
  • 08:26 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1283.eqiad.wmnet
  • 08:25 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1283.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 08:25 arnaudb@cumin1004: END (PASS) - Cookbook sre.gerrit.restart-gerrit (exit_code=0) Restarting Gerrit on gerrit2002
  • 08:23 arnaudb@cumin1004: START - Cookbook sre.gerrit.restart-gerrit Restarting Gerrit on gerrit2002
  • 08:22 arnaudb: gerrit2002: point the spare Gerrit at ldap-ro-test.codfw and restart it - T440888
  • 08:17 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2251.codfw.wmnet
  • 08:16 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2251.codfw.wmnet
  • 08:16 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2250.codfw.wmnet
  • 08:16 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1283.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 08:15 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1283.eqiad.wmnet
  • 08:15 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2250.codfw.wmnet
  • 08:15 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2250.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 08:15 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1283.eqiad.wmnet
  • 08:14 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1282.eqiad.wmnet
  • 08:14 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1282.eqiad.wmnet
  • 08:14 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1282.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 08:10 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2250.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 08:04 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1282.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 08:04 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1282.eqiad.wmnet
  • 08:03 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1282.eqiad.wmnet
  • 08:03 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1281.eqiad.wmnet
  • 08:01 arnaudb@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on gerrit2002.wikimedia.org with reason: testing
  • 08:01 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1281.eqiad.wmnet
  • 08:01 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1281.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 07:51 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1281.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 07:50 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1281.eqiad.wmnet
  • 07:50 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1281.eqiad.wmnet
  • 07:50 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1280.eqiad.wmnet
  • 07:47 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1280.eqiad.wmnet
  • 07:47 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1280.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 07:47 ayounsi@cumin1004: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox2003.codfw.wmnet,netbox1003.eqiad.wmnet with reason: Force redis 6.2.0 to fix regression - ayounsi@cumin1004
  • 07:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2250.codfw.wmnet
  • 07:45 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2250.codfw.wmnet
  • 07:45 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2249.codfw.wmnet
  • 07:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2249.codfw.wmnet
  • 07:44 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2249.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 07:43 awight: UTC morning deployment complete
  • 07:43 awight@deploy1003: Finished scap sync-world: Backport for Suppress language link notifications by default (T437577) (duration: 11m 14s)
  • 07:41 ayounsi@cumin1004: START - Cookbook sre.deploy.python-code netbox to netbox2003.codfw.wmnet,netbox1003.eqiad.wmnet with reason: Force redis 6.2.0 to fix regression - ayounsi@cumin1004
  • 07:38 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2249.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 07:38 awight@deploy1003: awight, seanleong-wmde: Continuing with deployment
  • 07:37 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1280.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 07:37 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1280.eqiad.wmnet
  • 07:36 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1280.eqiad.wmnet
  • 07:36 awight@deploy1003: awight, seanleong-wmde: Backport for Suppress language link notifications by default (T437577) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 07:36 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1279.eqiad.wmnet
  • 07:34 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1279.eqiad.wmnet
  • 07:33 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1279.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 07:32 awight@deploy1003: Started scap sync-world: Backport for Suppress language link notifications by default (T437577)
  • 07:30 mlitn@deploy1003: Finished scap sync-world: Backport for Refactor API stuff (T440847) (1.47.0-wmf.24) (duration: 10m 06s)
  • 07:24 mlitn@deploy1003: mlitn: Continuing with deployment
  • 07:24 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1279.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 07:24 mlitn@deploy1003: mlitn: Backport for Refactor API stuff (T440847) (1.47.0-wmf.24) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 07:24 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1279.eqiad.wmnet
  • 07:23 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1279.eqiad.wmnet
  • 07:23 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1278.eqiad.wmnet
  • 07:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1278.eqiad.wmnet
  • 07:21 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1278.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 07:20 mlitn@deploy1003: Started scap sync-world: Backport for Refactor API stuff (T440847) (1.47.0-wmf.24)
  • 07:15 wmde-fisch@deploy1003: Finished scap sync-world: Backport for Enable sub-references on enwiki (T440243) (duration: 13m 11s)
  • 07:11 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1278.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 07:11 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1278.eqiad.wmnet
  • 07:10 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1278.eqiad.wmnet
  • 07:10 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1277.eqiad.wmnet
  • 07:10 wmde-fisch@deploy1003: wmde-fisch: Continuing with deployment
  • 07:08 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1277.eqiad.wmnet
  • 07:08 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1277.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 07:07 wmde-fisch@deploy1003: wmde-fisch: Backport for Enable sub-references on enwiki (T440243) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 07:02 wmde-fisch@deploy1003: Started scap sync-world: Backport for Enable sub-references on enwiki (T440243)
  • 06:56 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1277.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 06:56 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1277.eqiad.wmnet
  • 06:55 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1277.eqiad.wmnet
  • 06:55 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1276.eqiad.wmnet
  • 06:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1276.eqiad.wmnet
  • 06:53 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1276.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 06:43 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1276.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 06:43 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1276.eqiad.wmnet
  • 06:42 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1276.eqiad.wmnet
  • 06:42 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1275.eqiad.wmnet
  • 06:42 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1275.eqiad.wmnet
  • 06:42 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1275.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 06:34 ayounsi@cumin1004: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Force redis 6.2.0 to fix regression - ayounsi@cumin1004
  • 06:33 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1275.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 06:33 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1275.eqiad.wmnet
  • 06:33 ayounsi@cumin1004: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Force redis 6.2.0 to fix regression - ayounsi@cumin1004
  • 06:33 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1275.eqiad.wmnet
  • 06:32 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1274.eqiad.wmnet
  • 06:32 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1274.eqiad.wmnet
  • 06:32 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1274.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 06:23 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1274.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 06:23 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1274.eqiad.wmnet
  • 06:22 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1274.eqiad.wmnet
  • 06:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1273.eqiad.wmnet
  • 06:22 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1273.eqiad.wmnet
  • 06:22 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1273.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 06:12 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1273.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 06:12 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1273.eqiad.wmnet
  • 06:10 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1273.eqiad.wmnet
  • 06:10 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1272.eqiad.wmnet
  • 06:10 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1272.eqiad.wmnet
  • 06:10 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1272.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 06:01 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1272.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 06:01 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1272.eqiad.wmnet
  • 06:00 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1272.eqiad.wmnet
  • 06:00 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1271.eqiad.wmnet
  • 06:00 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1271.eqiad.wmnet
  • 05:59 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1271.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 05:58 ayounsi@cumin1004: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox2003.codfw.wmnet,netbox1003.eqiad.wmnet with reason: Add PySocks and update wheels on netbox - ayounsi@cumin1004
  • 05:55 ayounsi@cumin1004: START - Cookbook sre.deploy.python-code netbox to netbox2003.codfw.wmnet,netbox1003.eqiad.wmnet with reason: Add PySocks and update wheels on netbox - ayounsi@cumin1004
  • 05:51 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1271.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 05:50 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1271.eqiad.wmnet
  • 05:50 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1271.eqiad.wmnet
  • 05:49 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1270.eqiad.wmnet
  • 05:49 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1270.eqiad.wmnet
  • 05:49 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1270.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 05:46 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 150506
  • 05:45 ayounsi@cumin1004: START - Cookbook sre.network.peering with action 'email' for AS: 150506
  • 05:45 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 53209
  • 05:45 ayounsi@cumin1004: START - Cookbook sre.network.peering with action 'email' for AS: 53209
  • 05:45 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 30781
  • 05:44 ayounsi@cumin1004: START - Cookbook sre.network.peering with action 'email' for AS: 30781
  • 05:43 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 28126
  • 05:42 ayounsi@cumin1004: START - Cookbook sre.network.peering with action 'email' for AS: 28126
  • 05:42 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 262731
  • 05:41 ayounsi@cumin1004: START - Cookbook sre.network.peering with action 'email' for AS: 262731
  • 05:41 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 44034
  • 05:41 ayounsi@cumin1004: START - Cookbook sre.network.peering with action 'email' for AS: 44034
  • 05:40 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 680
  • 05:40 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1270.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 05:40 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1270.eqiad.wmnet
  • 05:39 ayounsi@cumin1004: START - Cookbook sre.network.peering with action 'email' for AS: 680
  • 05:39 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1270.eqiad.wmnet
  • 05:39 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 30383
  • 05:39 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1269.eqiad.wmnet
  • 05:39 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1269.eqiad.wmnet
  • 05:39 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1269.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 05:39 ayounsi@cumin1004: START - Cookbook sre.network.peering with action 'email' for AS: 30383
  • 05:39 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 58495
  • 05:38 ayounsi@cumin1004: START - Cookbook sre.network.peering with action 'email' for AS: 58495
  • 05:38 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 8752
  • 05:38 ayounsi@cumin1004: START - Cookbook sre.network.peering with action 'email' for AS: 8752
  • 05:38 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 268937
  • 05:38 ayounsi@cumin1004: START - Cookbook sre.network.peering with action 'email' for AS: 268937
  • 05:37 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 9534
  • 05:36 ayounsi@cumin1004: START - Cookbook sre.network.peering with action 'email' for AS: 9534
  • 05:30 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1269.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 05:30 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1269.eqiad.wmnet
  • 05:30 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2249.codfw.wmnet
  • 05:29 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1269.eqiad.wmnet
  • 05:29 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2249.codfw.wmnet
  • 05:29 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1268.eqiad.wmnet
  • 05:29 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1268.eqiad.wmnet
  • 05:29 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2248.codfw.wmnet
  • 05:29 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2248.codfw.wmnet
  • 05:29 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1268.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 05:29 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2248.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 05:23 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2248.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 05:20 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1268.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 05:20 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1268.eqiad.wmnet
  • 05:17 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2248.codfw.wmnet
  • 05:16 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2248.codfw.wmnet
  • 05:16 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2243.codfw.wmnet
  • 05:15 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1268.eqiad.wmnet
  • 05:15 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1267.eqiad.wmnet
  • 05:15 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1267.eqiad.wmnet
  • 05:14 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2243.codfw.wmnet
  • 05:14 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2243.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 05:14 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1267.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 05:07 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2243.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 05:06 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2243.codfw.wmnet
  • 05:06 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2243.codfw.wmnet
  • 05:06 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2242.codfw.wmnet
  • 05:06 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2242.codfw.wmnet
  • 05:06 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1267.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 05:05 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2242.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 05:05 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1267.eqiad.wmnet
  • 05:05 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1267.eqiad.wmnet
  • 05:04 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1266.eqiad.wmnet
  • 05:04 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1266.eqiad.wmnet
  • 05:04 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1266.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 04:58 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2242.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 04:57 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2242.codfw.wmnet
  • 04:57 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2242.codfw.wmnet
  • 04:56 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2203.codfw.wmnet
  • 04:56 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2203.codfw.wmnet
  • 04:56 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2203.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 04:56 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1266.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 04:55 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1266.eqiad.wmnet
  • 04:55 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2203.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 04:55 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2203.codfw.wmnet
  • 04:55 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1266.eqiad.wmnet
  • 04:54 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1265.eqiad.wmnet
  • 04:54 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1265.eqiad.wmnet
  • 04:54 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2203.codfw.wmnet
  • 04:54 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1265.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 04:54 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2088.codfw.wmnet
  • 04:54 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2088.codfw.wmnet
  • 04:46 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1265.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 04:45 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1265.eqiad.wmnet
  • 04:45 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1265.eqiad.wmnet
  • 04:45 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1264.eqiad.wmnet
  • 04:45 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1264.eqiad.wmnet
  • 04:44 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1264.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 04:36 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1264.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 04:35 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1264.eqiad.wmnet
  • 04:35 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1264.eqiad.wmnet
  • 04:35 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1263.eqiad.wmnet
  • 04:35 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1263.eqiad.wmnet
  • 04:34 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1263.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 04:25 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1263.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 04:25 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1263.eqiad.wmnet
  • 04:24 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1263.eqiad.wmnet
  • 04:24 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1262.eqiad.wmnet
  • 04:24 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1262.eqiad.wmnet
  • 04:24 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1262.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 04:15 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1262.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 04:14 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1262.eqiad.wmnet
  • 04:14 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1262.eqiad.wmnet
  • 04:14 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1261.eqiad.wmnet
  • 04:13 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1261.eqiad.wmnet
  • 04:13 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1261.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 04:05 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1261.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 04:04 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1261.eqiad.wmnet
  • 04:04 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1261.eqiad.wmnet
  • 04:04 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1260.eqiad.wmnet
  • 04:04 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1260.eqiad.wmnet
  • 04:03 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1260.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 03:55 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1260.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 03:54 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1260.eqiad.wmnet
  • 03:54 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2088.codfw.wmnet
  • 03:54 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1260.eqiad.wmnet
  • 03:54 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D{wikikube-worker[1260-1289].eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad)
  • 03:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2088.codfw.wmnet
  • 03:53 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D{wikikube-worker[2088,2203,2242-2243,2248-2260].codfw.wmnet} and (A:wikikube-master-codfw or A:wikikube-worker-codfw)
  • 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 30s)
  • 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image
  • 00:51 dzahn@dns1004: END - running authdns-update
  • 00:48 dzahn@dns1004: START - running authdns-update
  • 00:45 brett@dns1004: END - running authdns-update
  • 00:43 brett@dns1004: START - running authdns-update

2026-10-13

  • 22:38 RoanKattouw: Deployed security patches for T440942, T440943, T441027 (at 22:22 UTC, forgot to log earlier)
  • 21:50 jdlrobson@deploy1003: Finished scap sync-world: Backport for EditWatchlist: use same view on mobile and desktop (T363032) (duration: 10m 37s)
  • 21:44 jdlrobson@deploy1003: jdlrobson: Continuing with deployment
  • 21:43 jdlrobson@deploy1003: jdlrobson: Backport for EditWatchlist: use same view on mobile and desktop (T363032) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 21:39 jdlrobson@deploy1003: Started scap sync-world: Backport for EditWatchlist: use same view on mobile and desktop (T363032)
  • 21:31 derenrich@deploy1003: Finished scap sync-world: Backport for Swap the preferred-sources experiment string to preferred-sources-ab (1.47.0-wmf.24), CentralNotice: Exclude the preferred sources experiment from banners (T436692) (duration: 12m 47s)
  • 21:26 derenrich@deploy1003: derenrich, egardner: Continuing with deployment
  • 21:23 derenrich@deploy1003: derenrich, egardner: Backport for Swap the preferred-sources experiment string to preferred-sources-ab (1.47.0-wmf.24), CentralNotice: Exclude the preferred sources experiment from banners (T436692) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 21:19 derenrich@deploy1003: Started scap sync-world: Backport for Swap the preferred-sources experiment string to preferred-sources-ab (1.47.0-wmf.24), CentralNotice: Exclude the preferred sources experiment from banners (T436692)
  • 21:16 jdlrobson@deploy1003: Finished scap sync-world: Backport for Hide "maybe later" button for campaign URL (T438251) (1.47.0-wmf.23), Re-Display success message if user has already consented (T438248) (1.47.0-wmf.23) (duration: 11m 42s)
  • 21:10 jdlrobson@deploy1003: jdlrobson: Continuing with deployment
  • 21:08 mutante: codesearch codesearch10 - systemctl restart hound-search
  • 21:08 jdlrobson@deploy1003: jdlrobson: Backport for Hide "maybe later" button for campaign URL (T438251) (1.47.0-wmf.23), Re-Display success message if user has already consented (T438248) (1.47.0-wmf.23) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 21:04 jdlrobson@deploy1003: Started scap sync-world: Backport for Hide "maybe later" button for campaign URL (T438251) (1.47.0-wmf.23), Re-Display success message if user has already consented (T438248) (1.47.0-wmf.23)
  • 20:42 aude@deploy1003: Finished scap sync-world: Backport for Load complete rows of cards on Special:ReadingLists (T440308) (1.47.0-wmf.23) (duration: 11m 47s)
  • 20:37 aude@deploy1003: aude: Continuing with deployment
  • 20:35 aude@deploy1003: aude: Backport for Load complete rows of cards on Special:ReadingLists (T440308) (1.47.0-wmf.23) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 20:34 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply
  • 20:33 eevans@deploy1003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply
  • 20:30 aude@deploy1003: Started scap sync-world: Backport for Load complete rows of cards on Special:ReadingLists (T440308) (1.47.0-wmf.23)
  • 20:28 krinkle@deploy1003: Finished scap sync-world: Backport for Profiler: Fold excimer line parsing into one method, Enable "TitleBlacklist" log channel (T85428) (duration: 11m 43s)
  • 20:23 krinkle@deploy1003: krinkle: Continuing with deployment
  • 20:21 krinkle@deploy1003: krinkle: Backport for Profiler: Fold excimer line parsing into one method, Enable "TitleBlacklist" log channel (T85428) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 20:17 krinkle@deploy1003: Started scap sync-world: Backport for Profiler: Fold excimer line parsing into one method, Enable "TitleBlacklist" log channel (T85428)
  • 20:12 kemayo@deploy1003: Finished scap sync-world: Backport for VisualEditor: Precompute edit suggestion counts on 5% of enwiki (T433024) (duration: 10m 28s)
  • 20:12 brett@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp2043.codfw.wmnet with reason: virtualization work
  • 20:09 jayme@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-main1010.eqiad.wmnet with OS trixie
  • 20:07 kemayo@deploy1003: kemayo: Continuing with deployment
  • 20:06 kemayo@deploy1003: kemayo: Backport for VisualEditor: Precompute edit suggestion counts on 5% of enwiki (T433024) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 20:02 kemayo@deploy1003: Started scap sync-world: Backport for VisualEditor: Precompute edit suggestion counts on 5% of enwiki (T433024)
  • 19:57 bking@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts krb1003.eqiad.wmnet
  • 19:57 bking@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 19:57 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: krb1003.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - bking@cumin2003"
  • 19:57 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: krb1003.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - bking@cumin2003"
  • 19:54 jayme@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-main1010.eqiad.wmnet with reason: host reimage
  • 19:51 jayme@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-main1010.eqiad.wmnet with reason: host reimage
  • 19:37 bking@cumin2003: START - Cookbook sre.dns.netbox
  • 19:35 jayme@cumin1004: START - Cookbook sre.hosts.reimage for host kafka-main1010.eqiad.wmnet with OS trixie
  • 19:32 bking@cumin2003: START - Cookbook sre.hosts.decommission for hosts krb1003.eqiad.wmnet
  • 19:30 dbrant@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply
  • 19:29 dbrant@deploy1003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply
  • 19:29 dbrant@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply
  • 19:29 dbrant@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply
  • 19:27 dbrant@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply
  • 19:27 dbrant@deploy1003: helmfile [staging] START helmfile.d/services/wikifeeds: apply
  • 19:26 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
  • 19:26 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
  • 19:26 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
  • 19:26 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
  • 19:22 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
  • 19:22 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
  • 19:22 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
  • 19:22 maryum: Deployed second security fix for T440944
  • 19:22 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
  • 19:14 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
  • 19:13 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
  • 19:13 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
  • 19:13 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
  • 19:12 maryum: Deployed security fix for T440944
  • 19:00 jayme@cumin1004: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['kafka-main1010.eqiad.wmnet']
  • 18:32 jayme@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['kafka-main1010.eqiad.wmnet']
  • 18:32 jayme@cumin1004: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['kafka-main1010.eqiad.wmnet']
  • 18:31 jayme@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['kafka-main1010.eqiad.wmnet']
  • 18:31 jayme@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['kafka-main1010.eqiad.wmnet']
  • 17:58 jayme@cumin1004: START - Cookbook sre.hosts.reimage for host kafka-main1010.eqiad.wmnet with OS trixie
  • 17:56 jayme@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host kafka-main1010.eqiad.wmnet with OS trixie
  • 17:48 jayme@cumin1004: START - Cookbook sre.hosts.reimage for host kafka-main1010.eqiad.wmnet with OS trixie
  • 17:41 btullis@cumin1004: START - Cookbook sre.ceph.rotate-osd-keys rolling rotate_keys on P{cephosd100[4-5].eqiad.wmnet} and (A:cephosd-codfw or A:cephosd-eqiad)
  • 17:37 btullis@cumin1004: END (PASS) - Cookbook sre.ceph.rotate-osd-keys (exit_code=0) rolling rotate_keys on P{cephosd1003.eqiad.wmnet} and (A:cephosd-codfw or A:cephosd-eqiad)
  • 17:25 isaranto@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
  • 17:14 isaranto@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
  • 17:07 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host kafka-main1010.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 17:06 elukey@cumin1004: START - Cookbook sre.hosts.provision for host kafka-main1010.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 17:04 btullis@cumin1004: START - Cookbook sre.ceph.rotate-osd-keys rolling rotate_keys on P{cephosd1003.eqiad.wmnet} and (A:cephosd-codfw or A:cephosd-eqiad)
  • 16:57 jayme@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on kafka-main1010.eqiad.wmnet with reason: firmware upgrade failed
  • 16:49 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.enable_virtualization (exit_code=0) rolling provision on D{wikikube-worker[2196-2215].codfw.wmnet} and (A:wikikube-master-codfw or A:wikikube-worker-codfw)
  • 16:49 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2215.codfw.wmnet
  • 16:48 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.enable_virtualization (exit_code=0) rolling provision on D{wikikube-worker[1240-1259].eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad)
  • 16:48 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1259.eqiad.wmnet
  • 16:47 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1259.eqiad.wmnet
  • 16:47 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1259.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 16:47 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2215.codfw.wmnet
  • 16:47 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2215.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 16:39 cmooney@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device ssw1-a1-eqiad
  • 16:38 cmooney@cumin1004: START - Cookbook sre.network.tls for network device ssw1-a1-eqiad
  • 16:37 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2215.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 16:37 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2215.codfw.wmnet
  • 16:37 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1259.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 16:37 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2215.codfw.wmnet
  • 16:37 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1259.eqiad.wmnet
  • 16:36 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2214.codfw.wmnet
  • 16:36 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2214.codfw.wmnet
  • 16:36 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2214.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 16:36 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1259.eqiad.wmnet
  • 16:36 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1258.eqiad.wmnet
  • 16:34 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1258.eqiad.wmnet
  • 16:33 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1258.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 16:26 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2214.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 16:26 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2214.codfw.wmnet
  • 16:25 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2214.codfw.wmnet
  • 16:25 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2213.codfw.wmnet
  • 16:23 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1258.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 16:23 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1258.eqiad.wmnet
  • 16:23 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2213.codfw.wmnet
  • 16:23 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2213.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 16:23 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1258.eqiad.wmnet
  • 16:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1257.eqiad.wmnet
  • 16:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1257.eqiad.wmnet
  • 16:20 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1257.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 16:20 jhathaway@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-cron: apply
  • 16:19 jhathaway@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-cron: apply
  • 16:18 btullis@cumin1004: END (FAIL) - Cookbook sre.ceph.rotate-osd-keys (exit_code=1) rolling rotate_keys on P{cephosd1003.eqiad.wmnet} and (A:cephosd-codfw or A:cephosd-eqiad)
  • 16:18 btullis@cumin1004: START - Cookbook sre.ceph.rotate-osd-keys rolling rotate_keys on P{cephosd1003.eqiad.wmnet} and (A:cephosd-codfw or A:cephosd-eqiad)
  • 16:13 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2213.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 16:13 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2213.codfw.wmnet
  • 16:13 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2213.codfw.wmnet
  • 16:12 brett@dns1004: END - running authdns-update
  • 16:12 jhathaway@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-cron: apply
  • 16:12 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2212.codfw.wmnet
  • 16:12 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2212.codfw.wmnet
  • 16:12 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2212.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 16:12 jhathaway@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-cron: apply
  • 16:10 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1257.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 16:10 brett@dns1004: START - running authdns-update
  • 16:10 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1257.eqiad.wmnet
  • 16:10 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1257.eqiad.wmnet
  • 16:09 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1256.eqiad.wmnet
  • 16:07 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1256.eqiad.wmnet
  • 16:07 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1256.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 16:03 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2212.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 16:02 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2212.codfw.wmnet
  • 16:02 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2212.codfw.wmnet
  • 16:01 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2211.codfw.wmnet
  • 16:01 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2211.codfw.wmnet
  • 16:01 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2211.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 15:57 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1256.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 15:57 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1256.eqiad.wmnet
  • 15:56 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1256.eqiad.wmnet
  • 15:56 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1255.eqiad.wmnet
  • 15:56 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1255.eqiad.wmnet
  • 15:56 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1255.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 15:55 dancy@deploy1003: Installation of scap version "4.297.0" completed for 158 hosts
  • 15:52 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2211.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 15:52 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2211.codfw.wmnet
  • 15:51 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2211.codfw.wmnet
  • 15:51 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2210.codfw.wmnet
  • 15:50 dancy@deploy1003: Installing scap version "4.297.0" for 158 host(s)
  • 15:49 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2210.codfw.wmnet
  • 15:49 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2210.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 15:46 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1255.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 15:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1255.eqiad.wmnet
  • 15:45 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1255.eqiad.wmnet
  • 15:45 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1254.eqiad.wmnet
  • 15:45 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1254.eqiad.wmnet
  • 15:44 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1254.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 15:41 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2210.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 15:41 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2210.codfw.wmnet
  • 15:40 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2210.codfw.wmnet
  • 15:40 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2209.codfw.wmnet
  • 15:40 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2209.codfw.wmnet
  • 15:39 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2209.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 15:36 oblivian@cumin1004: END (ERROR) - Cookbook sre.k8s.pool-depool-node (exit_code=97) pool for host wikikube-worker2192.codfw.wmnet
  • 15:36 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2192.codfw.wmnet
  • 15:34 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1254.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 15:33 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1254.eqiad.wmnet
  • 15:33 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1254.eqiad.wmnet
  • 15:33 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1253.eqiad.wmnet
  • 15:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1253.eqiad.wmnet
  • 15:31 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1253.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 15:30 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2209.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 15:30 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2209.codfw.wmnet
  • 15:29 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2209.codfw.wmnet
  • 15:29 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2208.codfw.wmnet
  • 15:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2208.codfw.wmnet
  • 15:27 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2208.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 15:23 herron@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-web: apply
  • 15:22 herron@deploy1003: helmfile [codfw] START helmfile.d/services/mw-web: apply
  • 15:22 herron@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-web: apply
  • 15:22 herron@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-web: apply
  • 15:21 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1253.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 15:20 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1253.eqiad.wmnet
  • 15:20 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1253.eqiad.wmnet
  • 15:20 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1252.eqiad.wmnet
  • 15:20 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1252.eqiad.wmnet
  • 15:19 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1252.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 15:18 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2208.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 15:17 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2208.codfw.wmnet
  • 15:17 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2208.codfw.wmnet
  • 15:17 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2207.codfw.wmnet
  • 15:14 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2207.codfw.wmnet
  • 15:14 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2207.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 15:09 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1252.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 15:09 jayme@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-main1010.eqiad.wmnet with reason: firmware upgrades
  • 15:09 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1252.eqiad.wmnet
  • 15:09 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1252.eqiad.wmnet
  • 15:08 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1251.eqiad.wmnet
  • 15:07 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1251.eqiad.wmnet
  • 15:06 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1251.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 15:05 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2207.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 15:05 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2207.codfw.wmnet
  • 15:04 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2207.codfw.wmnet
  • 15:04 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2206.codfw.wmnet
  • 15:04 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2206.codfw.wmnet
  • 15:04 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2206.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 14:57 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cp2043.codfw.wmnet
  • 14:57 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1251.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 14:56 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1251.eqiad.wmnet
  • 14:56 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1251.eqiad.wmnet
  • 14:55 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1250.eqiad.wmnet
  • 14:55 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1250.eqiad.wmnet
  • 14:55 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1250.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 14:55 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2206.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 14:55 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2206.codfw.wmnet
  • 14:54 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2206.codfw.wmnet
  • 14:54 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2205.codfw.wmnet
  • 14:54 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2205.codfw.wmnet
  • 14:54 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2205.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 14:49 cdobbins@cumin1004: START - Cookbook sre.hosts.reboot-single for host cp2043.codfw.wmnet
  • 14:48 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp2043.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 14:46 cdobbins@cumin1004: START - Cookbook sre.hosts.provision for host cp2043.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 14:45 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1250.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 14:45 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1250.eqiad.wmnet
  • 14:45 cdobbins@cumin1004: conftool action : set/pooled=no; selector: name=cp2043.*
  • 14:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1250.eqiad.wmnet
  • 14:44 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1249.eqiad.wmnet
  • 14:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1249.eqiad.wmnet
  • 14:44 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2205.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 14:44 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1249.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 14:44 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2205.codfw.wmnet
  • 14:43 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2205.codfw.wmnet
  • 14:43 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2204.codfw.wmnet
  • 14:42 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2204.codfw.wmnet
  • 14:42 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2204.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 14:38 ayounsi@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
  • 14:37 ayounsi@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
  • 14:34 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1249.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 14:33 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1249.eqiad.wmnet
  • 14:33 ayounsi@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary
  • 14:33 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1249.eqiad.wmnet
  • 14:33 ayounsi@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
  • 14:33 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2204.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 14:33 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1248.eqiad.wmnet
  • 14:32 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2204.codfw.wmnet
  • 14:32 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2204.codfw.wmnet
  • 14:32 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2203.codfw.wmnet
  • 14:32 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2203.codfw.wmnet
  • 14:31 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2203.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 14:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1248.eqiad.wmnet
  • 14:30 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen: apply
  • 14:30 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1248.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 14:30 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2203.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 14:30 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen: apply
  • 14:30 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2203.codfw.wmnet
  • 14:29 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2203.codfw.wmnet
  • 14:29 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2202.codfw.wmnet
  • 14:29 ayounsi@cumin1004: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Add PySocks and update wheels on netbox-next - ayounsi@cumin1004
  • 14:28 ayounsi@cumin1004: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Add PySocks and update wheels on netbox-next - ayounsi@cumin1004
  • 14:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2202.codfw.wmnet
  • 14:26 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2202.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 14:23 hashar@deploy1003: Finished deploy [gerrit/gerrit@3cd8cb0]: wm-pcc: switch commands from experimental to new puppet - T414621 (duration: 00m 20s)
  • 14:23 hashar@deploy1003: Started deploy [gerrit/gerrit@3cd8cb0]: wm-pcc: switch commands from experimental to new puppet - T414621
  • 14:20 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1248.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 14:20 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1248.eqiad.wmnet
  • 14:20 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1248.eqiad.wmnet
  • 14:19 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1247.eqiad.wmnet
  • 14:19 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1247.eqiad.wmnet
  • 14:19 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1247.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 14:18 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2202.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 14:17 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2202.codfw.wmnet
  • 14:17 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2202.codfw.wmnet
  • 14:17 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2201.codfw.wmnet
  • 14:14 kamila@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-syntaxhighlight: apply
  • 14:14 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2201.codfw.wmnet
  • 14:14 kamila@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-syntaxhighlight: apply
  • 14:14 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2201.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 14:11 kamila@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply
  • 14:10 kamila@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply
  • 14:10 kamila@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-syntaxhighlight: apply
  • 14:09 kamila@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-syntaxhighlight: apply
  • 14:09 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1247.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 14:09 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1247.eqiad.wmnet
  • 14:08 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1247.eqiad.wmnet
  • 14:08 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1246.eqiad.wmnet
  • 14:08 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1246.eqiad.wmnet
  • 14:08 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1246.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 14:05 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2201.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 14:05 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2201.codfw.wmnet
  • 14:04 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2201.codfw.wmnet
  • 14:04 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2200.codfw.wmnet
  • 14:04 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2200.codfw.wmnet
  • 14:04 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2200.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 14:02 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2004.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 14:01 elukey@cumin1004: START - Cookbook sre.hosts.provision for host sretest2004.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 14:01 btullis@cumin1004: END (FAIL) - Cookbook sre.ceph.rotate-osd-keys (exit_code=1) rolling rotate_keys on A:cephosd-eqiad
  • 13:58 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1246.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 13:57 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1246.eqiad.wmnet
  • 13:57 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1246.eqiad.wmnet
  • 13:57 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1245.eqiad.wmnet
  • 13:57 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1245.eqiad.wmnet
  • 13:56 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1245.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 13:54 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2200.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 13:54 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2200.codfw.wmnet
  • 13:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2200.codfw.wmnet
  • 13:53 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2199.codfw.wmnet
  • 13:51 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2199.codfw.wmnet
  • 13:51 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2199.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 13:49 kamila@deploy1003: Finished scap deploy-service shellbox: Update image version (duration: 11m 37s)
  • 13:48 ladsgroup@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2191: Maint done
  • 13:47 kamila@deploy1003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
  • 13:46 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1245.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 13:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1245.eqiad.wmnet
  • 13:45 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1245.eqiad.wmnet
  • 13:45 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1244.eqiad.wmnet
  • 13:45 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1244.eqiad.wmnet
  • 13:45 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1244.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 13:42 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2199.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 13:42 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2199.codfw.wmnet
  • 13:41 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2199.codfw.wmnet
  • 13:41 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2198.codfw.wmnet
  • 13:41 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2198.codfw.wmnet
  • 13:41 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2198.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 13:38 kamila@deploy1003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] codfw Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
  • 13:37 kamila@deploy1003: scap deploy-service shellbox: [STAGING/DEFAULT] staging-eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
  • 13:37 kamila@deploy1003: Started scap deploy-service shellbox: Update image version
  • 13:35 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1244.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 13:35 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1244.eqiad.wmnet
  • 13:34 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1244.eqiad.wmnet
  • 13:34 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1243.eqiad.wmnet
  • 13:33 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1243.eqiad.wmnet
  • 13:32 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1243.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 13:32 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2198.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 13:31 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2198.codfw.wmnet
  • 13:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2198.codfw.wmnet
  • 13:31 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2197.codfw.wmnet
  • 13:29 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2197.codfw.wmnet
  • 13:29 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2197.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 13:23 btullis@cumin1004: START - Cookbook sre.ceph.rotate-osd-keys rolling rotate_keys on A:cephosd-eqiad
  • 13:23 jayme@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'sync'.
  • 13:22 jayme@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'sync'.
  • 13:22 jayme@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'.
  • 13:22 jayme@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'.
  • 13:22 jayme@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'sync'.
  • 13:22 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1243.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 13:22 jayme@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'sync'.
  • 13:22 jayme@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'sync'.
  • 13:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1243.eqiad.wmnet
  • 13:22 jayme@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'sync'.
  • 13:22 jayme@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'.
  • 13:21 jayme@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'.
  • 13:21 jayme@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'.
  • 13:21 jayme@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'.
  • 13:21 jayme@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'.
  • 13:21 jayme@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'.
  • 13:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1243.eqiad.wmnet
  • 13:21 jayme@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'sync'.
  • 13:20 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1242.eqiad.wmnet
  • 13:20 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1242.eqiad.wmnet
  • 13:20 jayme@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'sync'.
  • 13:20 jayme@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'sync'.
  • 13:20 jayme@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'sync'.
  • 13:20 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1242.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 13:20 jayme@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'sync'.
  • 13:20 jelto@deploy1003: Finished scap deploy-service miscweb: test deployment of miscweb using deploy-service and spiderpig - annual report version bump (duration: 01m 54s)
  • 13:20 jayme@deploy1003: helmfile [codfw] START helmfile.d/admin 'sync'.
  • 13:20 jayme@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'sync'.
  • 13:20 jayme@deploy1003: helmfile [eqiad] START helmfile.d/admin 'sync'.
  • 13:19 jelto@deploy1003: scap deploy-service miscweb: [PRODUCTION/DEFAULT] eqiad Deploying miscweb
  • 13:19 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2197.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 13:19 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2197.codfw.wmnet
  • 13:19 jelto@deploy1003: scap deploy-service miscweb: [PRODUCTION/DEFAULT] codfw Deploying miscweb
  • 13:18 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2197.codfw.wmnet
  • 13:18 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2196.codfw.wmnet
  • 13:18 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2196.codfw.wmnet
  • 13:18 jelto@deploy1003: scap deploy-service miscweb: [STAGING/DEFAULT] staging-eqiad Deploying miscweb
  • 13:18 jelto@deploy1003: Started scap deploy-service miscweb: test deployment of miscweb using deploy-service and spiderpig - annual report version bump
  • 13:18 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2196.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 13:13 btullis@cumin1004: END (FAIL) - Cookbook sre.ceph.rotate-osd-keys (exit_code=1) rolling rotate_keys on A:cephosd-eqiad
  • 13:08 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2196.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 13:08 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1242.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 13:08 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2196.codfw.wmnet
  • 13:08 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1242.eqiad.wmnet
  • 13:07 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2196.codfw.wmnet
  • 13:07 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D{wikikube-worker[2196-2215].codfw.wmnet} and (A:wikikube-master-codfw or A:wikikube-worker-codfw)
  • 13:07 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1242.eqiad.wmnet
  • 13:07 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1241.eqiad.wmnet
  • 13:05 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1241.eqiad.wmnet
  • 13:05 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1241.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 13:03 ladsgroup@cumin1004: START - Cookbook sre.mysql.pool pool db2191: Maint done
  • 13:00 hashar@deploy1003: Finished scap sync-world: Backport for Revert "forms: Fix checkbox and radio misalignment on edit page" (T440971 T437803) (duration: 11m 24s)
  • 12:55 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1241.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 12:55 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1241.eqiad.wmnet
  • 12:54 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1241.eqiad.wmnet
  • 12:54 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1240.eqiad.wmnet
  • 12:54 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1240.eqiad.wmnet
  • 12:54 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1240.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 12:53 hashar@deploy1003: hashar: Continuing with deployment
  • 12:53 hashar@deploy1003: hashar: Backport for Revert "forms: Fix checkbox and radio misalignment on edit page" (T440971 T437803) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 12:49 ayounsi@cumin1004: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox2003.codfw.wmnet,netbox1003.eqiad.wmnet with reason: Add netbox-bgp and update wheels on netbox - ayounsi@cumin1004
  • 12:48 hashar@deploy1003: Started scap sync-world: Backport for Revert "forms: Fix checkbox and radio misalignment on edit page" (T440971 T437803)
  • 12:44 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1240.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 12:43 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1240.eqiad.wmnet
  • 12:38 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1240.eqiad.wmnet
  • 12:38 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D{wikikube-worker[1240-1259].eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad)
  • 12:38 ayounsi@cumin1004: START - Cookbook sre.deploy.python-code netbox to netbox2003.codfw.wmnet,netbox1003.eqiad.wmnet with reason: Add netbox-bgp and update wheels on netbox - ayounsi@cumin1004
  • 11:44 btullis@cumin1004: START - Cookbook sre.ceph.rotate-osd-keys rolling rotate_keys on A:cephosd-eqiad
  • 11:44 btullis@cumin1004: END (PASS) - Cookbook sre.ceph.rotate-osd-keys (exit_code=0) rolling rotate_keys on P{cephosd1001.eqiad.wmnet} and (A:cephosd-codfw or A:cephosd-eqiad)
  • 11:43 btullis@cumin1004: START - Cookbook sre.ceph.rotate-osd-keys rolling rotate_keys on P{cephosd1001.eqiad.wmnet} and (A:cephosd-codfw or A:cephosd-eqiad)
  • 11:43 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2191.codfw.wmnet with reason: Maintenance
  • 11:39 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2191 (T433990)', diff saved to https://phabricator.wikimedia.org/P96672 and previous config saved to /var/cache/conftool/dbconfig/20261013-113906-ladsgroup.json
  • 11:39 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2191.codfw.wmnet with reason: Maintenance
  • 11:30 kamila@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-syntaxhighlight: apply
  • 11:29 kamila@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-syntaxhighlight: apply
  • 11:29 kamila@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply
  • 11:29 kamila@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply
  • 11:26 kamila@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-syntaxhighlight: apply
  • 11:26 kamila@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-syntaxhighlight: apply
  • 11:13 hashar@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.47.0-wmf.24 refs T438220
  • 10:31 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on P{lvs1020.eqiad.wmnet} and A:lvs
  • 10:30 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on P{lvs1020.eqiad.wmnet} and A:lvs
  • 10:05 topranks: reboot ssw1-d1-eqiad to upgrade SR-Linux version to v26.7.1 T431745
  • 09:46 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 18 hosts with reason: upgrade ssw1-d1-eqiad
  • 09:28 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 18 hosts with reason: upgrade ssw1-d1-eqiad
  • 09:27 cmooney@cumin1004: DONE (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 1:00:00 on 20 hosts with reason: upgrade ssw1-d1-eqiad
  • 09:22 topranks: drain traffic from ssw1-d1-eqiad ahead of upgrade T431745
  • 09:20 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.enable_virtualization (exit_code=0) rolling provision on D{wikikube-worker[1140-1151,1153-1165].eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad)
  • 09:20 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1165.eqiad.wmnet
  • 09:20 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1165.eqiad.wmnet
  • 09:20 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1165.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 09:13 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1165.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 09:13 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1165.eqiad.wmnet
  • 09:12 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1165.eqiad.wmnet
  • 09:12 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1164.eqiad.wmnet
  • 09:12 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1164.eqiad.wmnet
  • 09:12 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1164.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 09:11 tappof@cumin1004: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling restart_daemons on A:kafka-logging-eqiad
  • 09:04 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1164.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 09:04 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1164.eqiad.wmnet
  • 09:03 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1164.eqiad.wmnet
  • 09:03 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1163.eqiad.wmnet
  • 09:03 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1163.eqiad.wmnet
  • 09:03 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1163.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 08:56 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1163.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 08:56 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1163.eqiad.wmnet
  • 08:55 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1163.eqiad.wmnet
  • 08:55 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1162.eqiad.wmnet
  • 08:55 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1162.eqiad.wmnet
  • 08:55 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1162.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 08:52 tappof@cumin1004: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling restart_daemons on A:kafka-logging-eqiad
  • 08:47 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1162.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 08:47 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1162.eqiad.wmnet
  • 08:46 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1162.eqiad.wmnet
  • 08:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1161.eqiad.wmnet
  • 08:46 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1161.eqiad.wmnet
  • 08:46 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1161.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 08:39 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1161.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 08:39 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1161.eqiad.wmnet
  • 08:34 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1161.eqiad.wmnet
  • 08:34 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1160.eqiad.wmnet
  • 08:34 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1160.eqiad.wmnet
  • 08:33 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1160.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 08:27 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1160.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 08:26 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1160.eqiad.wmnet
  • 08:26 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1160.eqiad.wmnet
  • 08:25 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1159.eqiad.wmnet
  • 08:25 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1159.eqiad.wmnet
  • 08:25 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1159.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 08:21 dpogorzelski@cumin1004: conftool action : set/pooled=true; selector: dnsdisc=k8s-ingress-ml-serve,name=eqiad
  • 08:21 dpogorzelski@cumin1004: conftool action : set/pooled=true; selector: dnsdisc=inference,name=eqiad
  • 08:18 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revertrisk' for release 'main' .
  • 08:18 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1159.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 08:17 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1159.eqiad.wmnet
  • 08:17 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1159.eqiad.wmnet
  • 08:16 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1158.eqiad.wmnet
  • 08:16 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'.
  • 08:15 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1158.eqiad.wmnet
  • 08:14 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1158.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 08:14 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'.
  • 08:12 dpogorzelski@cumin1004: conftool action : set/pooled=false; selector: dnsdisc=inference,name=eqiad
  • 08:12 dpogorzelski@cumin1004: conftool action : set/pooled=false; selector: dnsdisc=k8s-ingress-ml-serve,name=eqiad
  • 08:10 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
  • 08:09 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
  • 08:06 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1158.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 08:06 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1158.eqiad.wmnet
  • 08:05 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1158.eqiad.wmnet
  • 08:05 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1157.eqiad.wmnet
  • 08:05 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1157.eqiad.wmnet
  • 08:05 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1157.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 07:58 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1157.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 07:58 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1157.eqiad.wmnet
  • 07:58 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1157.eqiad.wmnet
  • 07:57 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1156.eqiad.wmnet
  • 07:57 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1156.eqiad.wmnet
  • 07:57 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1156.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 07:50 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1156.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 07:50 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1156.eqiad.wmnet
  • 07:49 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1156.eqiad.wmnet
  • 07:49 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1155.eqiad.wmnet
  • 07:49 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1155.eqiad.wmnet
  • 07:49 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1155.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 07:48 tappof@cumin1004: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling restart_daemons on A:kafka-logging-codfw
  • 07:42 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1155.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 07:41 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1155.eqiad.wmnet
  • 07:41 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1155.eqiad.wmnet
  • 07:41 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1154.eqiad.wmnet
  • 07:41 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1154.eqiad.wmnet
  • 07:41 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1154.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 07:33 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1154.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 07:33 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1154.eqiad.wmnet
  • 07:32 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1154.eqiad.wmnet
  • 07:32 wmde-fisch@deploy1003: Finished scap sync-world: Backport for kawiki: add temporary logo and icon for 200000 articles (T439994) (duration: 10m 51s)
  • 07:32 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1153.eqiad.wmnet
  • 07:32 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1153.eqiad.wmnet
  • 07:32 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1153.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 07:30 tappof@cumin1004: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling restart_daemons on A:kafka-logging-codfw
  • 07:27 wmde-fisch@deploy1003: wmde-fisch, anzx: Continuing with deployment
  • 07:25 wmde-fisch@deploy1003: wmde-fisch, anzx: Backport for kawiki: add temporary logo and icon for 200000 articles (T439994) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 07:23 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1153.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 07:23 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1153.eqiad.wmnet
  • 07:21 wmde-fisch@deploy1003: Started scap sync-world: Backport for kawiki: add temporary logo and icon for 200000 articles (T439994)
  • 07:18 wmde-fisch@deploy1003: Finished scap sync-world: Backport for Enable better autonames CommunityConfiguration everywhere (T439842) (duration: 14m 39s)
  • 07:18 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1153.eqiad.wmnet
  • 07:18 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1151.eqiad.wmnet
  • 07:18 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1151.eqiad.wmnet
  • 07:18 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1151.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 07:11 wmde-fisch@deploy1003: wmde-fisch: Continuing with deployment
  • 07:10 wmde-fisch@deploy1003: wmde-fisch: Backport for Enable better autonames CommunityConfiguration everywhere (T439842) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 07:10 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1151.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 07:09 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1151.eqiad.wmnet
  • 07:09 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1151.eqiad.wmnet
  • 07:09 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1150.eqiad.wmnet
  • 07:07 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1150.eqiad.wmnet
  • 07:06 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1150.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 07:04 wmde-fisch@deploy1003: Started scap sync-world: Backport for Enable better autonames CommunityConfiguration everywhere (T439842)
  • 06:58 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1150.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 06:58 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1150.eqiad.wmnet
  • 06:57 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1150.eqiad.wmnet
  • 06:57 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1149.eqiad.wmnet
  • 06:56 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1149.eqiad.wmnet
  • 06:55 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1149.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 06:47 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1149.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 06:47 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1149.eqiad.wmnet
  • 06:46 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1149.eqiad.wmnet
  • 06:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1148.eqiad.wmnet
  • 06:45 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1148.eqiad.wmnet
  • 06:44 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1148.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 06:44 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.enable_virtualization (exit_code=1) rolling provision on D{wikikube-worker[2184-2200].codfw.wmnet} and (A:wikikube-master-codfw or A:wikikube-worker-codfw)
  • 06:34 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1148.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 06:34 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1148.eqiad.wmnet
  • 06:34 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1148.eqiad.wmnet
  • 06:33 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1147.eqiad.wmnet
  • 06:32 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1147.eqiad.wmnet
  • 06:32 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1147.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 06:23 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1147.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 06:23 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1147.eqiad.wmnet
  • 06:22 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1147.eqiad.wmnet
  • 06:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1146.eqiad.wmnet
  • 06:22 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1146.eqiad.wmnet
  • 06:22 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1146.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 06:14 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1146.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 06:14 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1146.eqiad.wmnet
  • 06:13 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1146.eqiad.wmnet
  • 06:13 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1145.eqiad.wmnet
  • 06:13 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1145.eqiad.wmnet
  • 06:13 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1145.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 06:10 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2195.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 06:04 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1145.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 06:04 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1145.eqiad.wmnet
  • 06:03 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1145.eqiad.wmnet
  • 06:03 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1144.eqiad.wmnet
  • 06:01 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1144.eqiad.wmnet
  • 06:01 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1144.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 05:53 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1144.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 05:53 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1144.eqiad.wmnet
  • 05:52 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1144.eqiad.wmnet
  • 05:52 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1143.eqiad.wmnet
  • 05:52 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1143.eqiad.wmnet
  • 05:52 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1143.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 05:43 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1143.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 05:43 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1143.eqiad.wmnet
  • 05:43 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1143.eqiad.wmnet
  • 05:42 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1142.eqiad.wmnet
  • 05:42 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1142.eqiad.wmnet
  • 05:42 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1142.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 05:40 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2195.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 05:40 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2195.codfw.wmnet
  • 05:34 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2195.codfw.wmnet
  • 05:34 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2194.codfw.wmnet
  • 05:34 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2194.codfw.wmnet
  • 05:34 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1142.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 05:34 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2194.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 05:34 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1142.eqiad.wmnet
  • 05:33 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1142.eqiad.wmnet
  • 05:33 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1141.eqiad.wmnet
  • 05:33 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1141.eqiad.wmnet
  • 05:33 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1141.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 05:26 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1141.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 05:25 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2194.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 05:25 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1141.eqiad.wmnet
  • 05:25 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2194.codfw.wmnet
  • 05:25 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1141.eqiad.wmnet
  • 05:25 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D{wikikube-worker[1140-1151,1153-1165].eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad)
  • 05:20 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2194.codfw.wmnet
  • 05:20 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2193.codfw.wmnet
  • 05:19 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2193.codfw.wmnet
  • 05:18 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2193.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 05:09 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2193.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 05:08 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2193.codfw.wmnet
  • 05:08 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2193.codfw.wmnet
  • 05:07 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2192.codfw.wmnet
  • 04:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2192.codfw.wmnet
  • 04:01 mwpresync@deploy1003: Pruned MediaWiki: 1.47.0-wmf.21 (duration: 01m 02s)
  • 03:38 mwpresync@deploy1003: Finished scap sync-world: testwikis to 1.47.0-wmf.24 refs T438220 (duration: 35m 42s)
  • 03:03 mwpresync@deploy1003: Started scap sync-world: testwikis to 1.47.0-wmf.24 refs T438220
  • 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 20s)
  • 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image
  • 00:00 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
  • 00:00 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: sync

2026-10-12

  • 21:58 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/analytics-test: apply
  • 21:58 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/analytics-test: apply
  • 21:23 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
  • 21:22 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
  • 20:49 kemayo@deploy1003: Finished scap sync-world: Backport for Attribution: Pass the contributor counts URL template (T429834) (duration: 34m 41s)
  • 20:44 kemayo@deploy1003: pmiazga, kemayo: Continuing with deployment
  • 20:19 kemayo@deploy1003: pmiazga, kemayo: Backport for Attribution: Pass the contributor counts URL template (T429834) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 20:15 kemayo@deploy1003: Started scap sync-world: Backport for Attribution: Pass the contributor counts URL template (T429834)
  • 20:13 kemayo@deploy1003: Finished scap sync-world: Backport for VisualEditor: Precompute edit suggestion counts on 1% of enwiki (T433024) (duration: 10m 59s)
  • 20:08 kemayo@deploy1003: kemayo: Continuing with deployment
  • 20:07 kemayo@deploy1003: kemayo: Backport for VisualEditor: Precompute edit suggestion counts on 1% of enwiki (T433024) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 20:02 kemayo@deploy1003: Started scap sync-world: Backport for VisualEditor: Precompute edit suggestion counts on 1% of enwiki (T433024)
  • 19:18 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.enable_virtualization (exit_code=0) rolling provision on D{wikikube-worker[1121-1140].eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad)
  • 19:18 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1140.eqiad.wmnet
  • 19:18 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1140.eqiad.wmnet
  • 19:18 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1140.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 19:16 zabe: zabe@deploy1003:~$ mwscript updateSpecialPages.php commonswiki --only Wantedcategories # T437108
  • 19:14 musikanimal@deploy1003: Finished scap sync-world: Backport for CommonSettings: ensure 'usecodemirror' user option is unset globally (T440077) (duration: 13m 11s)
  • 19:12 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1140.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 19:11 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1140.eqiad.wmnet
  • 19:11 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1140.eqiad.wmnet
  • 19:11 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1139.eqiad.wmnet
  • 19:11 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1139.eqiad.wmnet
  • 19:10 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1139.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 19:09 musikanimal@deploy1003: musikanimal: Continuing with deployment
  • 19:07 musikanimal@deploy1003: musikanimal: Backport for CommonSettings: ensure 'usecodemirror' user option is unset globally (T440077) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 19:04 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1139.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 19:03 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1139.eqiad.wmnet
  • 19:03 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1139.eqiad.wmnet
  • 19:03 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1138.eqiad.wmnet
  • 19:03 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1138.eqiad.wmnet
  • 19:02 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1138.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 19:01 musikanimal@deploy1003: Started scap sync-world: Backport for CommonSettings: ensure 'usecodemirror' user option is unset globally (T440077)
  • 18:56 zabe@deploy1003: Finished scap sync-world: Backport for Re-enable WantedCategories on commons (T437108) (duration: 10m 33s)
  • 18:55 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1138.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 18:55 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1138.eqiad.wmnet
  • 18:51 zabe@deploy1003: zabe: Continuing with deployment
  • 18:50 zabe@deploy1003: zabe: Backport for Re-enable WantedCategories on commons (T437108) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 18:50 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1138.eqiad.wmnet
  • 18:49 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1137.eqiad.wmnet
  • 18:49 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1137.eqiad.wmnet
  • 18:49 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1137.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 18:46 zabe@deploy1003: Started scap sync-world: Backport for Re-enable WantedCategories on commons (T437108)
  • 18:43 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1137.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 18:43 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1137.eqiad.wmnet
  • 18:42 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1137.eqiad.wmnet
  • 18:42 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1136.eqiad.wmnet
  • 18:42 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1136.eqiad.wmnet
  • 18:41 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1136.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 18:35 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1136.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 18:34 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1136.eqiad.wmnet
  • 18:33 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1136.eqiad.wmnet
  • 18:33 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1135.eqiad.wmnet
  • 18:33 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1135.eqiad.wmnet
  • 18:33 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1135.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 18:26 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1135.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 18:26 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1135.eqiad.wmnet
  • 18:25 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1135.eqiad.wmnet
  • 18:25 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1134.eqiad.wmnet
  • 18:23 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1134.eqiad.wmnet
  • 18:23 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1134.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 18:15 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
  • 18:15 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1134.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 18:14 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1134.eqiad.wmnet
  • 18:14 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
  • 18:14 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1134.eqiad.wmnet
  • 18:14 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1133.eqiad.wmnet
  • 18:12 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1133.eqiad.wmnet
  • 18:12 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1133.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 18:04 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1133.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 18:04 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1133.eqiad.wmnet
  • 18:03 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1133.eqiad.wmnet
  • 18:03 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1132.eqiad.wmnet
  • 18:01 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1132.eqiad.wmnet
  • 18:01 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1132.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 17:57 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2192.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 17:53 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1132.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 17:52 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1132.eqiad.wmnet
  • 17:52 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1132.eqiad.wmnet
  • 17:52 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1131.eqiad.wmnet
  • 17:50 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1131.eqiad.wmnet
  • 17:50 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1131.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 17:41 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1131.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 17:41 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1131.eqiad.wmnet
  • 17:41 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1131.eqiad.wmnet
  • 17:40 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1130.eqiad.wmnet
  • 17:38 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1130.eqiad.wmnet
  • 17:38 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1130.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 17:30 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1130.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 17:29 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1130.eqiad.wmnet
  • 17:29 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1130.eqiad.wmnet
  • 17:29 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1129.eqiad.wmnet
  • 17:29 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1129.eqiad.wmnet
  • 17:29 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1129.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 17:26 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2192.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 17:26 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2192.codfw.wmnet
  • 17:25 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2192.codfw.wmnet
  • 17:25 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2191.codfw.wmnet
  • 17:25 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2191.codfw.wmnet
  • 17:25 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2191.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 17:20 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1129.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 17:20 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1129.eqiad.wmnet
  • 17:19 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1129.eqiad.wmnet
  • 17:19 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1128.eqiad.wmnet
  • 17:19 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1128.eqiad.wmnet
  • 17:19 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1128.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 17:16 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2191.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 17:15 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2191.codfw.wmnet
  • 17:15 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2191.codfw.wmnet
  • 17:15 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2190.codfw.wmnet
  • 17:13 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2190.codfw.wmnet
  • 17:12 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2190.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 17:11 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1128.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 17:10 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1128.eqiad.wmnet
  • 17:10 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1128.eqiad.wmnet
  • 17:10 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1127.eqiad.wmnet
  • 17:07 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1127.eqiad.wmnet
  • 17:07 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1127.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 17:07 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
  • 17:06 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
  • 17:04 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2190.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 17:03 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2190.codfw.wmnet
  • 17:03 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2190.codfw.wmnet
  • 17:02 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2189.codfw.wmnet
  • 17:01 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2189.codfw.wmnet
  • 17:00 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2189.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 16:59 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1127.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 16:59 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1127.eqiad.wmnet
  • 16:58 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1127.eqiad.wmnet
  • 16:58 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1126.eqiad.wmnet
  • 16:56 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1126.eqiad.wmnet
  • 16:56 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1126.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 16:51 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2189.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 16:50 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2189.codfw.wmnet
  • 16:50 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2189.codfw.wmnet
  • 16:50 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2188.codfw.wmnet
  • 16:50 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2188.codfw.wmnet
  • 16:49 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2188.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 16:47 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1126.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 16:47 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1126.eqiad.wmnet
  • 16:47 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1126.eqiad.wmnet
  • 16:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1125.eqiad.wmnet
  • 16:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1125.eqiad.wmnet
  • 16:44 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1125.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 16:39 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2188.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 16:39 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2188.codfw.wmnet
  • 16:38 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2188.codfw.wmnet
  • 16:38 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2187.codfw.wmnet
  • 16:38 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2187.codfw.wmnet
  • 16:38 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2187.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 16:37 logmsgbot: ladsgroup Deployed security patch for T440737
  • 16:36 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1125.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 16:36 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1125.eqiad.wmnet
  • 16:35 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1125.eqiad.wmnet
  • 16:35 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1124.eqiad.wmnet
  • 16:35 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1124.eqiad.wmnet
  • 16:35 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1124.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 16:29 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2187.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 16:29 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2187.codfw.wmnet
  • 16:29 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2187.codfw.wmnet
  • 16:28 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2186.codfw.wmnet
  • 16:27 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1124.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 16:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2186.codfw.wmnet
  • 16:27 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1124.eqiad.wmnet
  • 16:27 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2186.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 16:26 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1124.eqiad.wmnet
  • 16:26 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1123.eqiad.wmnet
  • 16:25 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1123.eqiad.wmnet
  • 16:24 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1123.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 16:20 logmsgbot: ladsgroup Deployed security patch for T440741
  • 16:19 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2186.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 16:18 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2186.codfw.wmnet
  • 16:18 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2186.codfw.wmnet
  • 16:18 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2185.codfw.wmnet
  • 16:16 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1123.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 16:16 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1123.eqiad.wmnet
  • 16:16 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2185.codfw.wmnet
  • 16:15 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2185.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 16:15 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1123.eqiad.wmnet
  • 16:15 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1122.eqiad.wmnet
  • 16:15 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1122.eqiad.wmnet
  • 16:15 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1122.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 16:11 jayme@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
  • 16:10 jayme@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
  • 16:08 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2185.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 16:08 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2185.codfw.wmnet
  • 16:07 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2185.codfw.wmnet
  • 16:07 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2184.codfw.wmnet
  • 16:07 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1122.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 16:06 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1122.eqiad.wmnet
  • 16:06 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1122.eqiad.wmnet
  • 16:06 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1121.eqiad.wmnet
  • 16:06 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1121.eqiad.wmnet
  • 16:05 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1121.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 16:05 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2184.codfw.wmnet
  • 16:05 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2184.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 16:02 jayme@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
  • 16:02 jayme@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply
  • 15:58 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1121.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 15:57 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1121.eqiad.wmnet
  • 15:57 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1121.eqiad.wmnet
  • 15:57 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D{wikikube-worker[1121-1140].eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad)
  • 15:56 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2184.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 15:56 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2184.codfw.wmnet
  • 15:55 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2184.codfw.wmnet
  • 15:55 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D{wikikube-worker[2184-2200].codfw.wmnet} and (A:wikikube-master-codfw or A:wikikube-worker-codfw)
  • 14:59 jayme@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
  • 14:58 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/spark-history: apply
  • 14:58 jayme@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply
  • 14:57 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/spark-history: apply
  • 14:53 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
  • 14:49 jayme@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'.
  • 14:48 jayme@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'.
  • 14:48 jayme@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
  • 14:48 jayme@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'.
  • 14:48 jayme@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'.
  • 14:47 jayme@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'.
  • 14:46 jayme@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
  • 14:46 jayme@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
  • 14:32 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/analytics-test: apply
  • 14:32 moritzm: installing cjose security updates
  • 14:31 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/analytics-test: apply
  • 14:18 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/spark-history: apply
  • 14:17 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/spark-history: apply
  • 14:12 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync
  • 14:11 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync
  • 13:54 awight: UTC afternoon backport complete
  • 13:53 awight@deploy1003: Finished scap sync-world: Backport for Ordered site export (T439101) (duration: 10m 40s)
  • 13:48 awight@deploy1003: awight: Continuing with deployment
  • 13:47 awight@deploy1003: awight: Backport for Ordered site export (T439101) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 13:45 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/spark-history: apply
  • 13:44 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/spark-history: apply
  • 13:43 awight@deploy1003: Started scap sync-world: Backport for Ordered site export (T439101)
  • 13:32 moritzm: installing Perl security updates
  • 13:31 osleger@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
  • 13:31 osleger@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
  • 13:31 osleger@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
  • 13:31 osleger@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
  • 13:28 osleger@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
  • 13:27 osleger@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
  • 13:27 osleger@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
  • 13:26 osleger@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
  • 13:21 Lucas_WMDE: UTC afternoon backport+config window done
  • 13:20 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for eowiki: Create templateeditor usergroup and protection level (T440764) (duration: 13m 07s)
  • 13:15 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, tryvix1509: Continuing with deployment
  • 13:11 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, tryvix1509: Backport for eowiki: Create templateeditor usergroup and protection level (T440764) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 13:09 jelto@cumin1004: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: version upgrade
  • 13:07 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for eowiki: Create templateeditor usergroup and protection level (T440764)
  • 12:57 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
  • 12:51 moritzm: installing glib2.0 security updates
  • 12:41 jelto@cumin1004: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: version upgrade
  • 12:39 jelto@cumin1004: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab2002.wikimedia.org with reason: version upgrade
  • 12:30 jelto@cumin1004: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab2002.wikimedia.org with reason: version upgrade
  • 12:28 jelto@cumin1004: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1003.wikimedia.org with reason: version upgrade
  • 12:27 dpogorzelski@cumin1004: conftool action : set/pooled=true; selector: dnsdisc=inference,name=codfw
  • 12:27 dpogorzelski@cumin1004: conftool action : set/pooled=true; selector: dnsdisc=k8s-ingress-ml-serve,name=codfw
  • 12:21 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'article-descriptions' for release 'main' .
  • 12:18 jelto@cumin1004: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: version upgrade
  • 12:03 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'.
  • 12:00 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'.
  • 12:00 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
  • 12:00 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
  • 11:52 dpogorzelski@cumin1004: conftool action : set/pooled=false; selector: dnsdisc=k8s-ingress-ml-serve,name=codfw
  • 11:50 dpogorzelski@cumin1004: conftool action : set/pooled=false; selector: dnsdisc=inference,name=codfw
  • 11:38 blake@deploy1003: Finished scap sync-world: enabling native k8s sidecars for canary mw-api-ext, mw-api-int, mw-jobrunner, and mw-web - T417800 (duration: 04m 32s)
  • 11:35 blake@deploy1003: Started scap sync-world: enabling native k8s sidecars for canary mw-api-ext, mw-api-int, mw-jobrunner, and mw-web - T417800
  • 11:33 bjensen: enabling native k8s sidecars for canary mw-api-ext, mw-api-int, mw-jobrunner, and mw-web (T417800)
  • 11:25 zabe@deploy1003: Finished scap sync-world: Backport for Move reads/writes of wbc_entity_usage to x1 for ruwiki (T439753), Simplify Hadoop configuration for query pages (T309738) (duration: 11m 58s)
  • 11:20 zabe@deploy1003: zabe: Continuing with deployment
  • 11:17 zabe@deploy1003: zabe: Backport for Move reads/writes of wbc_entity_usage to x1 for ruwiki (T439753), Simplify Hadoop configuration for query pages (T309738) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 11:15 topranks: add row a/b vlans on lsw1-d7-eqiad and lsw1-c7-eqiad to trunk to low-traffic/backup LVS hosts
  • 11:13 zabe@deploy1003: Started scap sync-world: Backport for Move reads/writes of wbc_entity_usage to x1 for ruwiki (T439753), Simplify Hadoop configuration for query pages (T309738)
  • 11:04 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.roll-restart-reboot-wikimedia-dns (exit_code=0) rolling restart_daemons on A:wikidough
  • 10:54 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
  • 10:52 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
  • 10:51 btullis: deleting obsolete spark images from our docker-registry for T438397
  • 10:51 topranks: restarting pdns on wikidough hosts to pick up change / fix dnssec validation
  • 10:50 cmooney@cumin1004: START - Cookbook sre.dns.roll-restart-reboot-wikimedia-dns rolling restart_daemons on A:wikidough
  • 10:45 sukhe@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for doh3006.wikimedia.org
  • 10:45 sukhe@cumin1004: START - Cookbook sre.hosts.remove-downtime for doh3006.wikimedia.org
  • 10:42 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
  • 10:40 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply
  • 10:38 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
  • 10:38 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply
  • 10:23 jayme@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'.
  • 10:22 jayme@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'.
  • 10:22 jayme@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'.
  • 10:22 jayme@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'.
  • 10:22 jayme@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'.
  • 10:22 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/lac-lambdas: apply
  • 10:22 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/lac-lambdas: apply
  • 10:22 jayme@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'.
  • 10:22 jayme@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
  • 10:21 jayme@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
  • 10:21 jayme@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'.
  • 10:20 jayme@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'.
  • 10:20 jayme@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'.
  • 10:20 jayme@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'.
  • 10:20 jayme@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'.
  • 10:18 jayme@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'.
  • 10:18 jayme@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'.
  • 10:17 jayme@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'.
  • 10:17 jayme@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
  • 10:17 jayme@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'.
  • 10:15 jayme@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'sync'.
  • 10:15 jayme@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'sync'.
  • 10:13 jayme@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'sync'.
  • 10:13 jayme@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'sync'.
  • 10:03 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/lac-lambdas: apply
  • 10:03 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/lac-lambdas: apply
  • 09:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2191 (T433990)', diff saved to https://phabricator.wikimedia.org/P96666 and previous config saved to /var/cache/conftool/dbconfig/20261012-090551-ladsgroup.json
  • 09:01 trueg@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply
  • 09:00 trueg@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply
  • 09:00 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply
  • 08:59 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply
  • 08:55 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2191', diff saved to https://phabricator.wikimedia.org/P96665 and previous config saved to /var/cache/conftool/dbconfig/20261012-085549-ladsgroup.json
  • 08:50 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
  • 08:48 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
  • 08:47 wmde-fisch@deploy1003: Finished scap sync-world: Backport for Enable better autonames CommunityConfiguration on group0 (T440357) (duration: 29m 00s)
  • 08:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2191', diff saved to https://phabricator.wikimedia.org/P96664 and previous config saved to /var/cache/conftool/dbconfig/20261012-084547-ladsgroup.json
  • 08:40 wmde-fisch@deploy1003: wmde-fisch: Continuing with deployment
  • 08:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2191 (T433990)', diff saved to https://phabricator.wikimedia.org/P96663 and previous config saved to /var/cache/conftool/dbconfig/20261012-083544-ladsgroup.json
  • 08:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2191 (T433990)', diff saved to https://phabricator.wikimedia.org/P96662 and previous config saved to /var/cache/conftool/dbconfig/20261012-083440-ladsgroup.json
  • 08:34 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2191.codfw.wmnet with reason: Maintenance
  • 08:22 wmde-fisch@deploy1003: wmde-fisch: Backport for Enable better autonames CommunityConfiguration on group0 (T440357) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 08:18 wmde-fisch@deploy1003: Started scap sync-world: Backport for Enable better autonames CommunityConfiguration on group0 (T440357)
  • 08:12 wmde-fisch@deploy1003: Finished scap sync-world: Backport for Switch to CommunityConfiguration to enable better autonames in Cite VE (T436255) (duration: 32m 32s)
  • 08:09 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wmde: apply
  • 08:08 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wmde: apply
  • 08:07 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wikidata: apply
  • 08:07 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wikidata: apply
  • 08:06 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-sre: apply
  • 08:06 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-sre: apply
  • 08:01 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-search: apply
  • 08:01 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-search: apply
  • 08:01 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-research: apply
  • 08:00 wmde-fisch@deploy1003: wmde-fisch: Continuing with deployment
  • 08:00 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-research: apply
  • 07:59 wmde-fisch@deploy1003: wmde-fisch: Backport for Switch to CommunityConfiguration to enable better autonames in Cite VE (T436255) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 07:59 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-platform-eng: apply
  • 07:59 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-platform-eng: apply
  • 07:58 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
  • 07:58 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply
  • 07:57 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply
  • 07:57 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply
  • 07:56 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-fr-tech: apply
  • 07:55 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-fr-tech: apply
  • 07:55 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-experiment-platform: apply
  • 07:54 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-experiment-platform: apply
  • 07:54 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
  • 07:54 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
  • 07:53 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply
  • 07:53 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply
  • 07:51 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-product: apply
  • 07:51 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-product: apply
  • 07:40 wmde-fisch@deploy1003: Started scap sync-world: Backport for Switch to CommunityConfiguration to enable better autonames in Cite VE (T436255)
  • 07:30 kartik@deploy1003: Finished scap sync-world: Backport for ULS: Define LanguageSelectorV2SupportedSkins config (T439707) (duration: 17m 45s)
  • 07:22 kartik@deploy1003: kartik, abi: Continuing with deployment
  • 07:16 kartik@deploy1003: kartik, abi: Backport for ULS: Define LanguageSelectorV2SupportedSkins config (T439707) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 07:12 kartik@deploy1003: Started scap sync-world: Backport for ULS: Define LanguageSelectorV2SupportedSkins config (T439707)
  • 07:11 ladsgroup@deploy1003: Finished scap sync-world: Backport for Enable blocked external domains on metawiki (T337431) (duration: 38m 01s)
  • 06:57 ladsgroup@deploy1003: ladsgroup: Continuing with deployment
  • 06:51 ladsgroup@deploy1003: ladsgroup: Backport for Enable blocked external domains on metawiki (T337431) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 06:50 jmm@cumin2003: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Scott French out of all services on: 2441 hosts
  • 06:32 ladsgroup@deploy1003: Started scap sync-world: Backport for Enable blocked external domains on metawiki (T337431)
  • 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 41s)
  • 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image
  • 00:00 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
  • 00:00 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
  • 00:00 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
  • 00:00 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: sync

2026-10-11

  • 20:06 sukhe@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on doh3006.wikimedia.org with reason: host is depooled
  • 16:05 ssastry@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
  • 16:05 ssastry@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
  • 16:05 ssastry@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
  • 16:05 ssastry@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
  • 16:03 ssastry@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
  • 16:02 ssastry@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
  • 16:02 ssastry@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
  • 16:02 ssastry@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
  • 13:36 aikochou@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
  • 12:43 aikochou@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
  • 12:00 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
  • 12:00 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
  • 12:00 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
  • 12:00 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: sync
  • 02:01 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 01m 04s)
  • 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image
  • 00:00 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
  • 00:00 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: sync

2026-10-10

  • 12:00 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
  • 12:00 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
  • 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 13s)
  • 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image
  • 00:00 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
  • 00:00 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
  • 00:00 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
  • 00:00 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: sync

2026-10-09

  • 18:54 cscott@deploy1003: Finished scap sync-world: Backport for Undeploy Parsoid readviews on frwikisource (T440695) (duration: 09m 58s)
  • 18:49 cscott@deploy1003: cscott, ssastry: Continuing with deployment
  • 18:48 cscott@deploy1003: cscott, ssastry: Backport for Undeploy Parsoid readviews on frwikisource (T440695) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 18:44 cscott@deploy1003: Started scap sync-world: Backport for Undeploy Parsoid readviews on frwikisource (T440695)
  • 17:38 Daimona: Run database query from T440557#12411029 in x1.wikishared
  • 16:47 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on netbox1003.eqiad.wmnet with reason: Restoring DB from backup on netboxdb1003
  • 16:05 cmooney@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
  • 15:50 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 15:50 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: merge new dns entries for eqiad ab links - cmooney@cumin1004"
  • 15:50 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: merge new dns entries for eqiad ab links - cmooney@cumin1004"
  • 15:46 cmooney@cumin1004: START - Cookbook sre.dns.netbox
  • 15:44 cmooney@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
  • 15:44 cmooney@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary
  • 15:42 cmooney@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
  • 15:42 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for wikikube-worker1081.eqiad.wmnet
  • 15:42 oblivian@cumin1004: START - Cookbook sre.hosts.remove-downtime for wikikube-worker1081.eqiad.wmnet
  • 15:42 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1081.eqiad.wmnet
  • 15:42 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1081.eqiad.wmnet
  • 15:42 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for wikikube-worker1057.eqiad.wmnet
  • 15:42 oblivian@cumin1004: START - Cookbook sre.hosts.remove-downtime for wikikube-worker1057.eqiad.wmnet
  • 14:40 fnegri@deploy1003: helmfile [eqiad] DONE helmfile.d/services/toolhub: apply
  • 14:40 fnegri@deploy1003: helmfile [eqiad] START helmfile.d/services/toolhub: apply
  • 14:40 fnegri@deploy1003: helmfile [codfw] DONE helmfile.d/services/toolhub: apply
  • 14:40 fnegri@deploy1003: helmfile [codfw] START helmfile.d/services/toolhub: apply
  • 14:39 fnegri@deploy1003: helmfile [staging] DONE helmfile.d/services/toolhub: apply
  • 14:39 fnegri@deploy1003: helmfile [staging] START helmfile.d/services/toolhub: apply
  • 14:33 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 14:32 elukey@cumin1004: START - Cookbook sre.hosts.provision for host sretest2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 14:32 elukey@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 14:30 trueg@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply
  • 14:27 trueg@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply
  • 14:25 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply
  • 14:24 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply
  • 14:20 ammarpad@deploy1003: mwscript-k8s job started: extensions/Translate/scripts/moveTranslatableBundle.php --wiki mediawikiwiki 'Abstract Wikipedia team' 'Abstract Wikipedia' Ammarpad --reason 'requested at T440606'
  • 14:16 elukey@cumin1004: START - Cookbook sre.hosts.provision for host sretest2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 13:59 elukey@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['sretest2003.codfw.wmnet']
  • 13:57 fnegri@deploy1003: helmfile [eqiad] DONE helmfile.d/services/toolhub: apply
  • 13:56 fnegri@deploy1003: helmfile [eqiad] START helmfile.d/services/toolhub: apply
  • 13:56 fnegri@deploy1003: helmfile [codfw] DONE helmfile.d/services/toolhub: apply
  • 13:55 fnegri@deploy1003: helmfile [codfw] START helmfile.d/services/toolhub: apply
  • 13:54 fnegri@deploy1003: helmfile [staging] DONE helmfile.d/services/toolhub: apply
  • 13:53 fnegri@deploy1003: helmfile [staging] START helmfile.d/services/toolhub: apply
  • 13:40 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.enable_virtualization (exit_code=0) rolling provision on D{wikikube-worker[1113-1120].eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad)
  • 13:40 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1120.eqiad.wmnet
  • 13:39 elukey@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['sretest2003.codfw.wmnet']
  • 13:38 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1120.eqiad.wmnet
  • 13:38 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1120.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 13:38 elukey@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 13:31 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1120.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 13:30 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1120.eqiad.wmnet
  • 13:30 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1120.eqiad.wmnet
  • 13:30 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1119.eqiad.wmnet
  • 13:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1119.eqiad.wmnet
  • 13:27 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1119.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 13:19 elukey@cumin1004: START - Cookbook sre.hosts.provision for host sretest2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 13:19 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1119.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 13:19 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1119.eqiad.wmnet
  • 13:18 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1119.eqiad.wmnet
  • 13:18 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1118.eqiad.wmnet
  • 13:18 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1118.eqiad.wmnet
  • 13:17 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1118.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 13:10 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1118.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 13:09 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1118.eqiad.wmnet
  • 13:09 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1118.eqiad.wmnet
  • 13:08 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1117.eqiad.wmnet
  • 13:07 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1117.eqiad.wmnet
  • 13:07 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1117.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 12:59 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on sretest1006.eqiad.wmnet with reason: need a few more things in place before we can reimage it in new location
  • 12:59 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1117.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 12:58 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 12:58 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update entries for sretest1006 - cmooney@cumin1004"
  • 12:58 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1117.eqiad.wmnet
  • 12:58 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update entries for sretest1006 - cmooney@cumin1004"
  • 12:58 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1117.eqiad.wmnet
  • 12:58 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1116.eqiad.wmnet
  • 12:56 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1116.eqiad.wmnet
  • 12:55 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1116.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 12:54 cmooney@cumin1004: START - Cookbook sre.dns.netbox
  • 12:47 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1116.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 12:47 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1116.eqiad.wmnet
  • 12:46 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1116.eqiad.wmnet
  • 12:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1115.eqiad.wmnet
  • 12:45 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1115.eqiad.wmnet
  • 12:44 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1115.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 12:36 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1115.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 12:36 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1115.eqiad.wmnet
  • 12:36 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1115.eqiad.wmnet
  • 12:35 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1114.eqiad.wmnet
  • 12:33 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1114.eqiad.wmnet
  • 12:33 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1114.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 12:31 jayme@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 14 days, 0:00:00 on wikikube-worker1081.eqiad.wmnet with reason: hardware issues
  • 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2374.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc
  • 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2373.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc
  • 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2372.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc
  • 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2371.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc
  • 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2370.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc
  • 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2369.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc
  • 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2368.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc
  • 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2367.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc
  • 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2366.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc
  • 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2365.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc
  • 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2364.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc
  • 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2363.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc
  • 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2362.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc
  • 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2361.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc
  • 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2360.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc
  • 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2359.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc
  • 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2358.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc
  • 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2357.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc
  • 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2356.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc
  • 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2355.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc
  • 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2354.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc
  • 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2353.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc
  • 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2352.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc
  • 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2351.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc
  • 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2350.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc
  • 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2349.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc
  • 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2348.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc
  • 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2347.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc
  • 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2346.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc
  • 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2345.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc
  • 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2344.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc
  • 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2343.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc
  • 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2342.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc
  • 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2341.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc
  • 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2340.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc
  • 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2339.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc
  • 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2338.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc
  • 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2337.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc
  • 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2336.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc
  • 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2335.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc
  • 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2334.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc
  • 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2333.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc
  • 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2332.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc
  • 12:28 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1374.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc
  • 12:28 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1373.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc
  • 12:28 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1370.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc
  • 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1369.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc
  • 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1368.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc
  • 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1367.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc
  • 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1366.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc
  • 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1365.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc
  • 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1364.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc
  • 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1363.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc
  • 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1362.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc
  • 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1347.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc
  • 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1334.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc
  • 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1333.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc
  • 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1332.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc
  • 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1331.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc
  • 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1330.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc
  • 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1329.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc
  • 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1328.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc
  • 12:26 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1328.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc
  • 12:25 jayme@cumin1004: conftool action : set/weight=10; selector: name=wikikube-worker1328.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc
  • 12:25 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1114.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 12:25 jayme@cumin1004: conftool action : set/weight=10; selector: name=wikikube-worker1328.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc
  • 12:25 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1114.eqiad.wmnet
  • 12:24 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1114.eqiad.wmnet
  • 12:24 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1113.eqiad.wmnet
  • 12:23 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1113.eqiad.wmnet
  • 12:22 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1113.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 12:14 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1113.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 12:14 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1113.eqiad.wmnet
  • 12:13 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1113.eqiad.wmnet
  • 12:13 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D{wikikube-worker[1113-1120].eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad)
  • 12:00 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
  • 12:00 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
  • 12:00 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
  • 12:00 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: sync
  • 11:58 aikochou@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
  • 11:56 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
  • 11:53 fceratto@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2902: test
  • 11:52 fceratto@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2901: test
  • 11:42 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
  • 11:38 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db2902: test
  • 11:37 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db2901: test
  • 11:35 fceratto@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1903: test
  • 11:35 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1903: test
  • 11:35 fceratto@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1902: test
  • 11:35 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1902: test
  • 11:33 fceratto@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1901: test
  • 11:33 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1901: test
  • 11:33 fceratto@cumin1004: END (PASS) - Cookbook sre.mysql.clone (exit_code=0) of db2901.codfw.wmnet onto db2902.codfw.wmnet
  • 11:07 gkyziridis@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' .
  • 10:54 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
  • 10:51 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
  • 10:47 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
  • 10:24 gkyziridis@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' .
  • 10:20 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host druid1010.eqiad.wmnet with OS bookworm
  • 10:17 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on sretest1006.eqiad.wmnet with reason: move sretest1006 to row A
  • 10:16 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on lswtest-d8-eqiad,lswtest-d8-eqiad IPv6,ssw1-d[1,8]-eqiad with reason: take rail kit from test switch
  • 10:05 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
  • 10:04 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
  • 10:04 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
  • 10:04 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
  • 10:02 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
  • 10:02 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cuminunpriv1001.eqiad.wmnet
  • 10:02 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 10:02 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cuminunpriv1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003"
  • 10:01 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cuminunpriv1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003"
  • 09:59 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
  • 09:58 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on druid1010.eqiad.wmnet with reason: host reimage
  • 09:57 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply
  • 09:53 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on druid1010.eqiad.wmnet with reason: host reimage
  • 09:52 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
  • 09:51 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply
  • 09:40 codders: Updated the Wikidata property suggester with data from 20261005's JSON dump
  • 09:37 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host druid1010.eqiad.wmnet with OS bookworm
  • 09:23 jmm@cumin2003: START - Cookbook sre.dns.netbox
  • 09:18 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cuminunpriv1001.eqiad.wmnet
  • 08:45 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
  • 08:43 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.enable_virtualization (exit_code=0) rolling provision on D{wikikube-worker[1084-1087,1093-1095].eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad)
  • 08:43 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1095.eqiad.wmnet
  • 08:41 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1095.eqiad.wmnet
  • 08:41 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1095.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 08:33 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1095.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 08:33 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1095.eqiad.wmnet
  • 08:32 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1095.eqiad.wmnet
  • 08:32 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1094.eqiad.wmnet
  • 08:30 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1094.eqiad.wmnet
  • 08:30 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1094.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 08:22 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1094.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 08:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1094.eqiad.wmnet
  • 08:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1094.eqiad.wmnet
  • 08:21 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1093.eqiad.wmnet
  • 08:18 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1093.eqiad.wmnet
  • 08:18 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1093.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 08:10 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1093.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 08:10 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1093.eqiad.wmnet
  • 08:09 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1093.eqiad.wmnet
  • 08:09 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1087.eqiad.wmnet
  • 08:09 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1087.eqiad.wmnet
  • 08:08 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1087.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 08:00 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1087.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 07:59 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1087.eqiad.wmnet
  • 07:59 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1087.eqiad.wmnet
  • 07:59 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1086.eqiad.wmnet
  • 07:57 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1086.eqiad.wmnet
  • 07:57 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1086.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 07:48 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1086.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 07:47 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1086.eqiad.wmnet
  • 07:47 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1086.eqiad.wmnet
  • 07:47 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1085.eqiad.wmnet
  • 07:47 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1085.eqiad.wmnet
  • 07:46 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1085.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 07:38 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1085.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 07:37 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1085.eqiad.wmnet
  • 07:37 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1085.eqiad.wmnet
  • 07:36 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1084.eqiad.wmnet
  • 07:34 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1084.eqiad.wmnet
  • 07:34 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1084.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 07:25 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1084.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 07:25 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1084.eqiad.wmnet
  • 07:24 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1084.eqiad.wmnet
  • 07:24 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D{wikikube-worker[1084-1087,1093-1095].eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad)
  • 06:24 oblivian@cumin1004: END (ERROR) - Cookbook sre.hardware.upgrade-firmware (exit_code=97) upgrade firmware for hosts wikikube-worker1081.eqiad.wmnet
  • 05:51 oblivian@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wikikube-worker1081.eqiad.wmnet
  • 05:50 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1057.eqiad.wmnet
  • 05:50 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1057.eqiad.wmnet
  • 05:45 oblivian@cumin1004: END (ERROR) - Cookbook sre.k8s.pool-depool-node (exit_code=97) pool for host wikikube-worker1057.eqiad.wmnet
  • 05:45 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1057.eqiad.wmnet
  • 05:42 oblivian@cumin1004: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts wikikube-worker1081.eqiad.wmnet
  • 05:42 oblivian@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wikikube-worker1081.eqiad.wmnet
  • 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 37s)
  • 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image
  • 01:52 ssastry@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
  • 01:52 ssastry@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
  • 01:52 ssastry@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
  • 01:51 ssastry@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
  • 01:30 kemayo@deploy1003: Finished scap sync-world: Backport for BaseEditCheck: don't use toDataElement when looking for categories (T440669) (duration: 10m 10s)
  • 01:25 kemayo@deploy1003: kemayo: Continuing with deployment
  • 01:24 kemayo@deploy1003: kemayo: Backport for BaseEditCheck: don't use toDataElement when looking for categories (T440669) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 01:20 kemayo@deploy1003: Started scap sync-world: Backport for BaseEditCheck: don't use toDataElement when looking for categories (T440669)
  • 01:14 mutante: releases1003 - /usr/sbin/sysctl -w net.ipv4.tcp_congestion_control=bbr T440068 - downloads 120x times faster

2026-10-08

  • 23:39 fceratto@cumin1004: END (FAIL) - Cookbook sre.mysql.pool (exit_code=99) pool db2902: Pool db2902.codfw.wmnet in after cloning
  • 23:28 ejegg@deploy1003: Finished scap sync-world: Backport for Update CentralNotice to cd3f46a4053cb74ff50b003e (duration: 37m 26s)
  • 23:16 ejegg@deploy1003: ejegg: Continuing with deployment
  • 23:11 ejegg@deploy1003: ejegg: Backport for Update CentralNotice to cd3f46a4053cb74ff50b003e synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 22:51 ejegg@deploy1003: Started scap sync-world: Backport for Update CentralNotice to cd3f46a4053cb74ff50b003e
  • 22:37 kemayo@deploy1003: Finished scap sync-world: Backport for VisualEditor: Precompute edit suggestion counts on testwiki (T432733) (duration: 36m 40s)
  • 22:25 kemayo@deploy1003: kemayo: Continuing with deployment
  • 22:22 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
  • 22:20 kemayo@deploy1003: kemayo: Backport for VisualEditor: Precompute edit suggestion counts on testwiki (T432733) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 22:20 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
  • 22:13 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
  • 22:11 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply
  • 22:04 herron@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply
  • 22:03 herron@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply
  • 22:03 herron@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply
  • 22:02 herron@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply
  • 22:00 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
  • 22:00 kemayo@deploy1003: Started scap sync-world: Backport for VisualEditor: Precompute edit suggestion counts on testwiki (T432733)
  • 22:00 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply
  • 21:42 ejegg@deploy1003: Finished scap sync-world: Backport for Update CentralNotice to 658422ff5fed886 (duration: 35m 22s)
  • 21:37 ejegg@deploy1003: ejegg: Rolling back deployment
  • 21:27 ejegg@deploy1003: ejegg: Backport for Update CentralNotice to 658422ff5fed886 synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 21:15 cmooney@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-c1-eqiad
  • 21:15 cmooney@cumin1004: START - Cookbook sre.network.tls for network device lsw1-c1-eqiad
  • 21:14 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply
  • 21:13 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply
  • 21:07 ejegg@deploy1003: Started scap sync-world: Backport for Update CentralNotice to 658422ff5fed886
  • 20:48 cjming@deploy1003: Finished scap sync-world: Backport for Override experiment enrollments via X-Wikimedia-Debug header (T437240) (duration: 09m 40s)
  • 20:43 cjming@deploy1003: cjming: Continuing with deployment
  • 20:42 cjming@deploy1003: cjming: Backport for Override experiment enrollments via X-Wikimedia-Debug header (T437240) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 20:38 cjming@deploy1003: Started scap sync-world: Backport for Override experiment enrollments via X-Wikimedia-Debug header (T437240)
  • 20:33 mlitn@deploy1003: Finished scap sync-world: Backport for Temporary: don't catch exception (duration: 09m 41s)
  • 20:27 mlitn@deploy1003: mlitn: Continuing with deployment
  • 20:27 mlitn@deploy1003: mlitn: Backport for Temporary: don't catch exception synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 20:23 mlitn@deploy1003: Started scap sync-world: Backport for Temporary: don't catch exception
  • 20:21 cjming@deploy1003: Finished scap sync-world: Backport for Add URLs for MobileAppRedirect Special Page (T434930), Update Test Kitchen instruments intake URL on beta cluster (T437720) (duration: 14m 56s)
  • 20:14 cjming@deploy1003: milazg, cjming: Continuing with deployment
  • 20:10 cjming@deploy1003: milazg, cjming: Backport for Add URLs for MobileAppRedirect Special Page (T434930), Update Test Kitchen instruments intake URL on beta cluster (T437720) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 20:06 cjming@deploy1003: Started scap sync-world: Backport for Add URLs for MobileAppRedirect Special Page (T434930), Update Test Kitchen instruments intake URL on beta cluster (T437720)
  • 19:58 kamila@deploy1003: Finished scap deploy-service shellbox: PHP update (duration: 36m 05s)
  • 19:57 kamila@deploy1003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
  • 19:26 kamila@deploy1003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] codfw Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
  • 19:22 kamila@deploy1003: scap deploy-service shellbox: [STAGING/DEFAULT] staging-eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
  • 19:22 kamila@deploy1003: Started scap deploy-service shellbox: PHP update
  • 19:07 dzahn@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
  • 19:07 dzahn@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
  • 19:07 dzahn@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
  • 19:07 dzahn@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
  • 19:06 dzahn@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
  • 19:06 dzahn@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
  • 19:02 andrew@deploy1003: helmfile [eqiad] DONE helmfile.d/services/developer-portal: apply
  • 19:02 andrew@deploy1003: helmfile [eqiad] START helmfile.d/services/developer-portal: apply
  • 19:01 andrew@deploy1003: helmfile [codfw] DONE helmfile.d/services/developer-portal: apply
  • 19:01 andrew@deploy1003: helmfile [codfw] START helmfile.d/services/developer-portal: apply
  • 19:00 andrew@deploy1003: helmfile [staging] DONE helmfile.d/services/developer-portal: apply
  • 19:00 andrew@deploy1003: helmfile [staging] START helmfile.d/services/developer-portal: apply
  • 18:56 oblivian@cumin1004: END (ERROR) - Cookbook sre.hardware.upgrade-firmware (exit_code=97) upgrade firmware for hosts wikikube-worker1081.eqiad.wmnet
  • 17:47 andrew@deploy1003: helmfile [eqiad] DONE helmfile.d/services/developer-portal: apply
  • 17:46 andrew@deploy1003: helmfile [eqiad] START helmfile.d/services/developer-portal: apply
  • 17:46 andrew@deploy1003: helmfile [codfw] DONE helmfile.d/services/developer-portal: apply
  • 17:45 andrew@deploy1003: helmfile [codfw] START helmfile.d/services/developer-portal: apply
  • 17:36 andrew@deploy1003: helmfile [staging] DONE helmfile.d/services/developer-portal: apply
  • 17:36 andrew@deploy1003: helmfile [staging] START helmfile.d/services/developer-portal: apply
  • 17:31 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
  • 17:20 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.enable_virtualization (exit_code=0) rolling provision on D{wikikube-worker[2141-2179].codfw.wmnet} and (A:wikikube-master-codfw or A:wikikube-worker-codfw)
  • 17:20 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2179.codfw.wmnet
  • 17:20 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2179.codfw.wmnet
  • 17:10 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply
  • 17:09 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply
  • 17:09 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2179.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 17:04 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2179.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 17:03 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2179.codfw.wmnet
  • 17:03 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2179.codfw.wmnet
  • 17:03 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2178.codfw.wmnet
  • 17:02 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2178.codfw.wmnet
  • 16:49 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2178.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 16:44 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2178.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 16:44 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2178.codfw.wmnet
  • 16:43 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2178.codfw.wmnet
  • 16:43 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2177.codfw.wmnet
  • 16:43 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2177.codfw.wmnet
  • 16:38 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker1057.eqiad.wmnet
  • 16:37 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1057.eqiad.wmnet
  • 16:35 btullis@cumin1004: END (FAIL) - Cookbook sre.druid.reboot-workers (exit_code=99) for Druid public cluster: Reboot Druid nodes
  • 16:33 oblivian@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wikikube-worker1081.eqiad.wmnet
  • 16:28 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.enable_virtualization (exit_code=1) rolling provision on D{wikikube-worker[1064-1081,1084-1087,1093-1095].eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad)
  • 16:26 ssastry@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
  • 16:25 ssastry@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
  • 16:25 ssastry@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
  • 16:25 ssastry@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
  • 16:21 dancy@deploy1003: Installation of scap version "4.296.0" completed for 3 hosts
  • 16:19 dancy@deploy1003: Installing scap version "4.296.0" for 3 host(s)
  • 16:16 cmooney@cumin1004: END (FAIL) - Cookbook sre.network.tls (exit_code=99) for network device lsw1-c1-eqiad
  • 16:15 cmooney@cumin1004: START - Cookbook sre.network.tls for network device lsw1-c1-eqiad
  • 16:14 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 16:14 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add new dns entries for IPs on lsw1-c1-eqiad - cmooney@cumin1004"
  • 16:13 swfrench-wmf: begin rolling restarts of confd in ulsfo, eqsin, codfw to pick up etcd client SRV change
  • 16:10 elukey@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=1) for host sretest2009.codfw.wmnet with OS trixie
  • 16:07 swfrench@dns1004: END - running authdns-update
  • 16:04 swfrench@dns1004: START - running authdns-update
  • 15:56 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2009.codfw.wmnet (live debug)
  • 15:55 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest2009.codfw.wmnet (live debug)
  • 15:53 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2009.codfw.wmnet with reason: host reimage
  • 15:52 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1081.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 15:47 elukey@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2009.codfw.wmnet with reason: host reimage
  • 15:36 elukey@cumin1004: START - Cookbook sre.hosts.reimage for host sretest2009.codfw.wmnet with OS trixie
  • 15:29 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add new dns entries for IPs on lsw1-c1-eqiad - cmooney@cumin1004"
  • 15:25 cmooney@cumin1004: START - Cookbook sre.dns.netbox
  • 15:22 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1081.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 15:21 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1081.eqiad.wmnet
  • 15:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1081.eqiad.wmnet
  • 15:20 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1080.eqiad.wmnet
  • 15:18 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1080.eqiad.wmnet
  • 15:17 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1080.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 15:17 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db2902: Pool db2902.codfw.wmnet in after cloning
  • 15:09 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1080.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 15:08 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1080.eqiad.wmnet
  • 15:08 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1080.eqiad.wmnet
  • 15:07 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1079.eqiad.wmnet
  • 15:06 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1079.eqiad.wmnet
  • 15:06 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1079.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 14:59 btullis@cumin1004: START - Cookbook sre.druid.reboot-workers for Druid public cluster: Reboot Druid nodes
  • 14:58 aikochou@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
  • 14:57 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1079.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 14:57 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1079.eqiad.wmnet
  • 14:56 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1079.eqiad.wmnet
  • 14:56 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1078.eqiad.wmnet
  • 14:54 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1078.eqiad.wmnet
  • 14:54 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1078.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 14:54 cmooney@dns3003: END - running authdns-update
  • 14:51 cmooney@dns3003: START - running authdns-update
  • 14:50 cgoubert@dns2004: END - running authdns-update
  • 14:48 cgoubert@dns2004: START - running authdns-update
  • 14:48 moritzm: installing pyasn1 security updates
  • 14:45 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1078.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 14:45 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1078.eqiad.wmnet
  • 14:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1078.eqiad.wmnet
  • 14:44 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1077.eqiad.wmnet
  • 14:43 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 14:43 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add new dns entries for IPs on lsw1-c1-eqiad - cmooney@cumin1004"
  • 14:43 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add new dns entries for IPs on lsw1-c1-eqiad - cmooney@cumin1004"
  • 14:42 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1077.eqiad.wmnet
  • 14:42 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1077.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 14:39 cmooney@cumin1004: START - Cookbook sre.dns.netbox
  • 14:39 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 14:39 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add new dns entries for IPs on lsw1-c1-eqiad - cmooney@cumin1004"
  • 14:35 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add new dns entries for IPs on lsw1-c1-eqiad - cmooney@cumin1004"
  • 14:35 jdrewniak@deploy1003: Finished scap sync-world: Backport for Bumping portals to master (T128546) (duration: 36m 09s)
  • 14:33 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1077.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 14:32 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1077.eqiad.wmnet
  • 14:32 cmooney@cumin1004: START - Cookbook sre.dns.netbox
  • 14:32 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1077.eqiad.wmnet
  • 14:32 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1076.eqiad.wmnet
  • 14:32 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1076.eqiad.wmnet
  • 14:32 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1076.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 14:29 jdrewniak@deploy1003: jdrewniak: Continuing with deployment
  • 14:23 moritzm: installing perl security updates
  • 14:22 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1076.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 14:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1076.eqiad.wmnet
  • 14:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1076.eqiad.wmnet
  • 14:21 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1075.eqiad.wmnet
  • 14:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1075.eqiad.wmnet
  • 14:20 brouberol@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'.
  • 14:19 brouberol@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'.
  • 14:18 brouberol@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'.
  • 14:17 brouberol@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'.
  • 14:17 fceratto@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2901: Pool db2901.codfw.wmnet in after cloning
  • 14:09 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db2901: Pool db2901.codfw.wmnet in after cloning
  • 14:06 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1075.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 14:03 fceratto@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2902: Depool db2902.codfw.wmnet - fceratto@cumin1004
  • 14:03 fceratto@cumin1004: START - Cookbook sre.mysql.depool depool db2902: Depool db2902.codfw.wmnet - fceratto@cumin1004
  • 14:03 fceratto@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2901: Depool db2901.codfw.wmnet to then clone it to db2902.codfw.wmnet - fceratto@cumin1004
  • 14:03 fceratto@cumin1004: START - Cookbook sre.mysql.depool depool db2901: Depool db2901.codfw.wmnet to then clone it to db2902.codfw.wmnet - fceratto@cumin1004
  • 14:03 fceratto@cumin1004: START - Cookbook sre.mysql.clone of db2901.codfw.wmnet onto db2902.codfw.wmnet
  • 14:03 jdrewniak@deploy1003: jdrewniak: Backport for Bumping portals to master (T128546) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 13:58 jdrewniak@deploy1003: Started scap sync-world: Backport for Bumping portals to master (T128546)
  • 13:57 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1075.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 13:56 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1075.eqiad.wmnet
  • 13:56 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1075.eqiad.wmnet
  • 13:56 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1074.eqiad.wmnet
  • 13:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1074.eqiad.wmnet
  • 13:53 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1074.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 13:47 samtar@deploy1003: Finished scap sync-world: Backport for Home Experience: Feature flag to enable Home (T439738) (duration: 38m 16s)
  • 13:47 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
  • 13:44 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1074.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 13:44 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1074.eqiad.wmnet
  • 13:43 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1074.eqiad.wmnet
  • 13:43 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1073.eqiad.wmnet
  • 13:42 samtar@deploy1003: emc-wmf, samtar: Continuing with deployment
  • 13:41 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1073.eqiad.wmnet
  • 13:41 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1073.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 13:38 fceratto@cumin1004: END (FAIL) - Cookbook sre.mysql.clone (exit_code=99) of db2901.codfw.wmnet onto db2902.codfw.wmnet
  • 13:38 fceratto@cumin1004: START - Cookbook sre.mysql.clone of db2901.codfw.wmnet onto db2902.codfw.wmnet
  • 13:33 fceratto@cumin1004: END (FAIL) - Cookbook sre.mysql.clone (exit_code=99) of db2901.codfw.wmnet onto db2902.codfw.wmnet
  • 13:33 fceratto@cumin1004: START - Cookbook sre.mysql.clone of db2901.codfw.wmnet onto db2902.codfw.wmnet
  • 13:32 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1073.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 13:31 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1073.eqiad.wmnet
  • 13:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1073.eqiad.wmnet
  • 13:31 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1072.eqiad.wmnet
  • 13:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1072.eqiad.wmnet
  • 13:30 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1072.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 13:28 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2001.codfw.wmnet (live debug)
  • 13:27 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest2001.codfw.wmnet (live debug)
  • 13:22 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1072.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 13:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1072.eqiad.wmnet
  • 13:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1072.eqiad.wmnet
  • 13:21 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1071.eqiad.wmnet
  • 13:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1071.eqiad.wmnet
  • 13:20 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1071.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 13:13 samtar@deploy1003: emc-wmf, samtar: Backport for Home Experience: Feature flag to enable Home (T439738) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 13:13 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1071.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 13:12 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1071.eqiad.wmnet
  • 13:12 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1071.eqiad.wmnet
  • 13:12 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1070.eqiad.wmnet
  • 13:12 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1070.eqiad.wmnet
  • 13:11 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1070.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 13:09 samtar@deploy1003: Started scap sync-world: Backport for Home Experience: Feature flag to enable Home (T439738)
  • 13:05 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2177.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 13:04 fceratto@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1901: test
  • 13:04 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1901: test
  • 13:04 fceratto@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1901: test
  • 13:03 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1070.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 13:03 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1070.eqiad.wmnet
  • 13:03 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1070.eqiad.wmnet
  • 13:02 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1069.eqiad.wmnet
  • 13:02 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1069.eqiad.wmnet
  • 13:02 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1069.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 13:00 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2177.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 12:59 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2177.codfw.wmnet
  • 12:59 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2177.codfw.wmnet
  • 12:59 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2176.codfw.wmnet
  • 12:58 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2176.codfw.wmnet
  • 12:58 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2176.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 12:55 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1901: test
  • 12:54 fceratto@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1901: test
  • 12:54 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1901: test
  • 12:54 fceratto@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1901: test
  • 12:54 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1901: test
  • 12:54 fceratto@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1901: test
  • 12:54 fceratto@cumin1004: START - Cookbook sre.mysql.depool depool db1901: test
  • 12:54 fceratto@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1901: test
  • 12:54 dbrant@deploy1003: helmfile [codfw] DONE helmfile.d/services/mobileapps: apply
  • 12:53 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1069.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 12:53 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1069.eqiad.wmnet
  • 12:53 dbrant@deploy1003: helmfile [codfw] START helmfile.d/services/mobileapps: apply
  • 12:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1069.eqiad.wmnet
  • 12:53 dbrant@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply
  • 12:53 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1068.eqiad.wmnet
  • 12:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1068.eqiad.wmnet
  • 12:52 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1068.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 12:52 dbrant@deploy1003: helmfile [eqiad] START helmfile.d/services/mobileapps: apply
  • 12:51 dbrant@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
  • 12:50 dbrant@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
  • 12:50 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2176.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 12:49 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2176.codfw.wmnet
  • 12:49 zabe: zabe@deploy1003:~$ mwscript recountCategories.php commonswiki --mode all # T440023 T439902
  • 12:48 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2176.codfw.wmnet
  • 12:48 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2175.codfw.wmnet
  • 12:48 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2175.codfw.wmnet
  • 12:47 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1901: test
  • 12:44 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1068.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 12:44 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1068.eqiad.wmnet
  • 12:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1068.eqiad.wmnet
  • 12:43 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1067.eqiad.wmnet
  • 12:43 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1067.eqiad.wmnet
  • 12:43 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1067.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 12:39 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
  • 12:36 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2175.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 12:35 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1067.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 12:35 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1067.eqiad.wmnet
  • 12:34 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1067.eqiad.wmnet
  • 12:34 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1066.eqiad.wmnet
  • 12:32 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1066.eqiad.wmnet
  • 12:32 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1066.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 12:31 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2175.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 12:30 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2175.codfw.wmnet
  • 12:30 zabe@deploy1003: Finished scap sync-world: Backport for Use Hadoop for UnusedTemplates on commonswiki (T422443) (duration: 09m 47s)
  • 12:30 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2175.codfw.wmnet
  • 12:29 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2174.codfw.wmnet
  • 12:29 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2174.codfw.wmnet
  • 12:25 zabe@deploy1003: zabe: Continuing with deployment
  • 12:24 zabe@deploy1003: zabe: Backport for Use Hadoop for UnusedTemplates on commonswiki (T422443) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 12:23 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1066.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 12:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1066.eqiad.wmnet
  • 12:22 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1066.eqiad.wmnet
  • 12:21 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1065.eqiad.wmnet
  • 12:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1065.eqiad.wmnet
  • 12:21 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1065.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 12:20 zabe@deploy1003: Started scap sync-world: Backport for Use Hadoop for UnusedTemplates on commonswiki (T422443)
  • 12:18 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2174.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 12:13 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2174.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 12:13 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2174.codfw.wmnet
  • 12:12 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2174.codfw.wmnet
  • 12:12 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1065.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 12:12 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2173.codfw.wmnet
  • 12:12 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2173.codfw.wmnet
  • 12:12 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1065.eqiad.wmnet
  • 12:11 ihurbain@deploy1003: helmfile [codfw] DONE helmfile.d/services/mobileapps: apply
  • 12:11 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1065.eqiad.wmnet
  • 12:11 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1064.eqiad.wmnet
  • 12:10 ihurbain@deploy1003: helmfile [codfw] START helmfile.d/services/mobileapps: apply
  • 12:10 ihurbain@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply
  • 12:09 ihurbain@deploy1003: helmfile [eqiad] START helmfile.d/services/mobileapps: apply
  • 12:09 ihurbain@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
  • 12:08 ihurbain@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
  • 12:08 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1064.eqiad.wmnet
  • 12:08 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1064.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 12:00 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2173.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 12:00 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
  • 12:00 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
  • 11:59 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1064.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 11:59 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1064.eqiad.wmnet
  • 11:58 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1064.eqiad.wmnet
  • 11:58 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D{wikikube-worker[1064-1081,1084-1087,1093-1095].eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad)
  • 11:55 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2173.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 11:55 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2173.codfw.wmnet
  • 11:54 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2173.codfw.wmnet
  • 11:54 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2172.codfw.wmnet
  • 11:54 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2172.codfw.wmnet
  • 11:44 oblivian@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 14 days, 0:00:00 on wikikube-worker1057.eqiad.wmnet with reason: HW failure
  • 11:43 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2172.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 11:42 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply
  • 11:41 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply
  • 11:41 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply
  • 11:40 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply
  • 11:39 blake@deploy1003: Finished scap sync-world: update mcrouter exporter template for T417800 (duration: 04m 15s)
  • 11:38 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2172.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 11:37 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2172.codfw.wmnet
  • 11:37 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2172.codfw.wmnet
  • 11:36 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2171.codfw.wmnet
  • 11:36 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2171.codfw.wmnet
  • 11:36 blake@deploy1003: Started scap sync-world: update mcrouter exporter template for T417800
  • 11:24 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2171.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 11:19 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2171.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 11:19 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2171.codfw.wmnet
  • 11:18 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2171.codfw.wmnet
  • 11:18 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2170.codfw.wmnet
  • 11:18 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2170.codfw.wmnet
  • 11:18 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2170.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 11:11 aikochou@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
  • 11:10 topranks: cr1-codfw switch back to master routing engine T426197
  • 11:07 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2170.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 11:07 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2170.codfw.wmnet
  • 11:06 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2170.codfw.wmnet
  • 11:06 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2169.codfw.wmnet
  • 11:06 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2169.codfw.wmnet
  • 11:06 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2169.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 10:57 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.enable_virtualization (exit_code=1) rolling provision on D{wikikube-worker[1006-1007,1015-1016,1021,1034-1057].eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad)
  • 10:55 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2169.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 10:54 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2169.codfw.wmnet
  • 10:54 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2169.codfw.wmnet
  • 10:54 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2168.codfw.wmnet
  • 10:52 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2168.codfw.wmnet
  • 10:51 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2168.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 10:41 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2168.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 10:41 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2168.codfw.wmnet
  • 10:40 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2168.codfw.wmnet
  • 10:40 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2167.codfw.wmnet
  • 10:40 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2167.codfw.wmnet
  • 10:39 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2167.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 10:36 topranks: cr1-codfw switch to backup routing engine T426197
  • 10:31 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthbook: apply
  • 10:30 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook: apply
  • 10:29 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2167.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 10:29 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2167.codfw.wmnet
  • 10:28 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2167.codfw.wmnet
  • 10:28 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2166.codfw.wmnet
  • 10:26 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2166.codfw.wmnet
  • 10:26 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2166.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 10:19 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1262: Pooling back after cloning a host
  • 10:15 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2166.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 10:15 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2166.codfw.wmnet
  • 10:14 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2166.codfw.wmnet
  • 10:14 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2165.codfw.wmnet
  • 10:14 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2165.codfw.wmnet
  • 10:14 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2165.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 10:08 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2001.codfw.wmnet (live debug)
  • 10:06 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest2001.codfw.wmnet (live debug)
  • 10:04 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2165.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 10:03 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2165.codfw.wmnet
  • 10:03 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2165.codfw.wmnet
  • 10:02 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2164.codfw.wmnet
  • 10:02 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2164.codfw.wmnet
  • 10:02 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2164.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 09:59 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
  • 09:52 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 16 hosts with reason: cr1-codfw upgrade
  • 09:52 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2164.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 09:52 cmooney@cumin1004: DONE (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on 16 hosts with reason: cr1-codfw upgrade
  • 09:52 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2164.codfw.wmnet
  • 09:51 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2164.codfw.wmnet
  • 09:51 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2163.codfw.wmnet
  • 09:49 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2163.codfw.wmnet
  • 09:48 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2163.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 09:38 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2191: Repooling after switchover
  • 09:37 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2163.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 09:37 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2163.codfw.wmnet
  • 09:37 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2163.codfw.wmnet
  • 09:37 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2162.codfw.wmnet
  • 09:35 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2162.codfw.wmnet
  • 09:34 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2162.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 09:34 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db1262: Pooling back after cloning a host
  • 09:29 topranks: begin draining traffic from cr1-codfw in advance of JunOS upgrade T417871
  • 09:24 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2162.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 09:23 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2162.codfw.wmnet
  • 09:23 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2162.codfw.wmnet
  • 09:23 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2161.codfw.wmnet
  • 09:23 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2161.codfw.wmnet
  • 09:22 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2161.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 09:16 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1057.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 09:12 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2161.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 09:11 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2161.codfw.wmnet
  • 09:11 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2161.codfw.wmnet
  • 09:10 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2160.codfw.wmnet
  • 09:10 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2160.codfw.wmnet
  • 09:10 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2160.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 09:00 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2160.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 08:59 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2160.codfw.wmnet
  • 08:59 moritzm: installing Wireshark security updates
  • 08:59 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2160.codfw.wmnet
  • 08:59 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2159.codfw.wmnet
  • 08:57 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2159.codfw.wmnet
  • 08:57 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2159.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 08:53 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db2191: Repooling after switchover
  • 08:53 marostegui@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2191: Repooling after switchover
  • 08:53 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db2191: Repooling after switchover
  • 08:52 marostegui@cumin1004: dbctl commit (dc=all): 'Depool db2191 T436922', diff saved to https://phabricator.wikimedia.org/P96627 and previous config saved to /var/cache/conftool/dbconfig/20261008-085238-marostegui.json
  • 08:50 marostegui@cumin1004: dbctl commit (dc=all): 'Promote db2249 to x1 primary T436922', diff saved to https://phabricator.wikimedia.org/P96626 and previous config saved to /var/cache/conftool/dbconfig/20261008-085050-marostegui.json
  • 08:50 marostegui: Starting x1 codfw failover from db2191 to db2249 - T436922
  • 08:46 marostegui@cumin1004: dbctl commit (dc=all): 'Set db2249 with weight 0 T436922', diff saved to https://phabricator.wikimedia.org/P96625 and previous config saved to /var/cache/conftool/dbconfig/20261008-084653-marostegui.json
  • 08:46 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 17 hosts with reason: Primary switchover x1 T436922
  • 08:46 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
  • 08:46 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
  • 08:46 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1057.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 08:46 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
  • 08:46 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: sync
  • 08:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1057.eqiad.wmnet
  • 08:46 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2159.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 08:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2159.codfw.wmnet
  • 08:45 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2159.codfw.wmnet
  • 08:45 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2158.codfw.wmnet
  • 08:45 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1057.eqiad.wmnet
  • 08:44 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1056.eqiad.wmnet
  • 08:43 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2158.codfw.wmnet
  • 08:42 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2158.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 08:42 moritzm: installing python3.13 security updates
  • 08:42 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1056.eqiad.wmnet
  • 08:42 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1056.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 08:33 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1056.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 08:33 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1056.eqiad.wmnet
  • 08:32 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1056.eqiad.wmnet
  • 08:32 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1055.eqiad.wmnet
  • 08:32 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1055.eqiad.wmnet
  • 08:32 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1055.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 08:24 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1055.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 08:23 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1055.eqiad.wmnet
  • 08:23 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1055.eqiad.wmnet
  • 08:23 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1054.eqiad.wmnet
  • 08:22 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1054.eqiad.wmnet
  • 08:22 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1054.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 08:20 aklapper@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.47.0-wmf.23 refs T438219
  • 08:14 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1054.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 08:14 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1054.eqiad.wmnet
  • 08:13 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1054.eqiad.wmnet
  • 08:13 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1053.eqiad.wmnet
  • 08:13 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1053.eqiad.wmnet
  • 08:12 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1053.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 08:04 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1053.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 08:04 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1053.eqiad.wmnet
  • 08:03 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1053.eqiad.wmnet
  • 08:03 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1052.eqiad.wmnet
  • 08:03 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1052.eqiad.wmnet
  • 08:03 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1052.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 08:01 moritzm: installing mbedtls security updates
  • 07:55 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1052.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 07:55 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1052.eqiad.wmnet
  • 07:54 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1052.eqiad.wmnet
  • 07:54 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1051.eqiad.wmnet
  • 07:54 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1051.eqiad.wmnet
  • 07:53 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1051.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 07:46 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2158.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 07:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2158.codfw.wmnet
  • 07:46 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1051.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 07:45 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2158.codfw.wmnet
  • 07:45 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1051.eqiad.wmnet
  • 07:45 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2157.codfw.wmnet
  • 07:45 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2157.codfw.wmnet
  • 07:45 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2157.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 07:45 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1051.eqiad.wmnet
  • 07:44 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1050.eqiad.wmnet
  • 07:42 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1050.eqiad.wmnet
  • 07:42 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1050.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 07:34 ladsgroup@deploy1003: Finished scap sync-world: Backport for nrwiki, xhwiki: update logo and tagline (T440138 T440139) (duration: 13m 44s)
  • 07:34 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2157.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 07:34 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2157.codfw.wmnet
  • 07:33 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2157.codfw.wmnet
  • 07:33 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2156.codfw.wmnet
  • 07:33 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2156.codfw.wmnet
  • 07:33 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2156.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 07:32 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1050.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 07:32 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1050.eqiad.wmnet
  • 07:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1050.eqiad.wmnet
  • 07:31 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1049.eqiad.wmnet
  • 07:29 ladsgroup@deploy1003: anzx, ladsgroup: Continuing with deployment
  • 07:28 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1049.eqiad.wmnet
  • 07:28 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1049.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 07:25 ladsgroup@deploy1003: anzx, ladsgroup: Backport for nrwiki, xhwiki: update logo and tagline (T440138 T440139) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 07:22 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2156.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 07:21 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2156.codfw.wmnet
  • 07:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2156.codfw.wmnet
  • 07:20 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2155.codfw.wmnet
  • 07:20 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2155.codfw.wmnet
  • 07:20 ladsgroup@deploy1003: Started scap sync-world: Backport for nrwiki, xhwiki: update logo and tagline (T440138 T440139)
  • 07:20 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2155.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 07:19 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1049.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 07:19 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1049.eqiad.wmnet
  • 07:18 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1049.eqiad.wmnet
  • 07:18 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1048.eqiad.wmnet
  • 07:17 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1048.eqiad.wmnet
  • 07:16 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1048.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 07:14 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'.
  • 07:13 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'.
  • 07:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
  • 07:12 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
  • 07:12 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1265.eqiad.wmnet with reason: preparing backups
  • 07:09 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2155.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 07:09 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2155.codfw.wmnet
  • 07:08 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2155.codfw.wmnet
  • 07:08 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2154.codfw.wmnet
  • 07:07 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1048.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 07:07 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1048.eqiad.wmnet
  • 07:07 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2154.codfw.wmnet
  • 07:07 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1048.eqiad.wmnet
  • 07:06 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2154.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 07:06 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1047.eqiad.wmnet
  • 07:05 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1047.eqiad.wmnet
  • 07:04 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1047.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 06:56 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2154.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 06:55 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2154.codfw.wmnet
  • 06:55 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1047.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 06:55 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1047.eqiad.wmnet
  • 06:55 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2154.codfw.wmnet
  • 06:55 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2153.codfw.wmnet
  • 06:54 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1047.eqiad.wmnet
  • 06:54 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1046.eqiad.wmnet
  • 06:54 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1046.eqiad.wmnet
  • 06:54 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1046.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 06:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2153.codfw.wmnet
  • 06:52 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2153.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 06:45 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1046.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 06:45 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1046.eqiad.wmnet
  • 06:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1046.eqiad.wmnet
  • 06:44 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1045.eqiad.wmnet
  • 06:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1045.eqiad.wmnet
  • 06:44 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1045.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 06:41 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2153.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 06:41 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2153.codfw.wmnet
  • 06:41 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2153.codfw.wmnet
  • 06:40 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2152.codfw.wmnet
  • 06:39 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2152.codfw.wmnet
  • 06:39 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2152.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 06:37 Amir1: mwscript-k8s --follow --dblist=ores -- extensions/ORES/maintenance/PurgeScoreCache.php --model articlequality --old (T440358)
  • 06:35 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1045.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 06:35 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1045.eqiad.wmnet
  • 06:34 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1045.eqiad.wmnet
  • 06:34 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1044.eqiad.wmnet
  • 06:32 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1044.eqiad.wmnet
  • 06:31 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1044.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 06:28 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2152.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 06:27 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2152.codfw.wmnet
  • 06:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2152.codfw.wmnet
  • 06:27 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2151.codfw.wmnet
  • 06:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2151.codfw.wmnet
  • 06:26 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2151.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 06:23 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1044.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 06:23 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1044.eqiad.wmnet
  • 06:23 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1044.eqiad.wmnet
  • 06:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1043.eqiad.wmnet
  • 06:21 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
  • 06:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1043.eqiad.wmnet
  • 06:21 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1043.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 06:19 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
  • 06:15 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2151.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 06:15 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2151.codfw.wmnet
  • 06:15 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2151.codfw.wmnet
  • 06:14 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2150.codfw.wmnet
  • 06:13 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2150.codfw.wmnet
  • 06:13 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1043.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 06:13 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2150.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 06:12 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1043.eqiad.wmnet
  • 06:12 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1043.eqiad.wmnet
  • 06:12 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1042.eqiad.wmnet
  • 06:12 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
  • 06:10 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1042.eqiad.wmnet
  • 06:10 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1042.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 06:09 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply
  • 06:02 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1262.eqiad.wmnet with reason: preparing backups
  • 06:02 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2150.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 06:02 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1262: preparing backups
  • 06:02 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1042.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 06:02 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2150.codfw.wmnet
  • 06:02 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool db1262: preparing backups
  • 06:02 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1042.eqiad.wmnet
  • 06:01 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2150.codfw.wmnet
  • 06:01 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2149.codfw.wmnet
  • 06:01 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1042.eqiad.wmnet
  • 06:01 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2149.codfw.wmnet
  • 06:01 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1041.eqiad.wmnet
  • 06:01 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2149.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 06:00 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1041.eqiad.wmnet
  • 05:59 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1041.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 05:59 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
  • 05:58 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply
  • 05:51 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1041.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 05:51 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1041.eqiad.wmnet
  • 05:50 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1041.eqiad.wmnet
  • 05:50 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1040.eqiad.wmnet
  • 05:50 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2149.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 05:49 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2149.codfw.wmnet
  • 05:49 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1040.eqiad.wmnet
  • 05:49 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2149.codfw.wmnet
  • 05:49 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1040.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 05:49 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2148.codfw.wmnet
  • 05:46 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2148.codfw.wmnet
  • 05:46 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2148.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 05:39 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1040.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 05:39 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1040.eqiad.wmnet
  • 05:38 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1040.eqiad.wmnet
  • 05:38 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1039.eqiad.wmnet
  • 05:36 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2148.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 05:36 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1039.eqiad.wmnet
  • 05:35 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2148.codfw.wmnet
  • 05:35 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1039.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 05:35 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2148.codfw.wmnet
  • 05:35 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2147.codfw.wmnet
  • 05:33 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2147.codfw.wmnet
  • 05:32 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2147.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 05:28 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1039.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 05:27 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1039.eqiad.wmnet
  • 05:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1039.eqiad.wmnet
  • 05:26 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1038.eqiad.wmnet
  • 05:25 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1038.eqiad.wmnet
  • 05:24 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1038.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 05:22 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2147.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 05:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2147.codfw.wmnet
  • 05:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2147.codfw.wmnet
  • 05:21 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2146.codfw.wmnet
  • 05:19 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2146.codfw.wmnet
  • 05:19 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2146.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 05:16 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1038.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 05:16 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1038.eqiad.wmnet
  • 05:15 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1038.eqiad.wmnet
  • 05:15 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1037.eqiad.wmnet
  • 05:15 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1037.eqiad.wmnet
  • 05:15 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1037.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 05:08 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2146.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 05:08 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1037.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 05:08 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2146.codfw.wmnet
  • 05:08 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1037.eqiad.wmnet
  • 05:07 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2146.codfw.wmnet
  • 05:07 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2145.codfw.wmnet
  • 05:07 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1037.eqiad.wmnet
  • 05:07 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1036.eqiad.wmnet
  • 05:07 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1036.eqiad.wmnet
  • 05:07 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1036.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 05:05 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2145.codfw.wmnet
  • 05:05 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2145.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 04:59 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1036.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 04:59 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1036.eqiad.wmnet
  • 04:58 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1036.eqiad.wmnet
  • 04:58 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1035.eqiad.wmnet
  • 04:57 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1035.eqiad.wmnet
  • 04:56 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1035.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 04:54 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2145.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 04:54 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2145.codfw.wmnet
  • 04:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2145.codfw.wmnet
  • 04:53 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2144.codfw.wmnet
  • 04:52 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2144.codfw.wmnet
  • 04:51 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2144.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 04:47 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1035.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 04:47 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1035.eqiad.wmnet
  • 04:46 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1035.eqiad.wmnet
  • 04:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1034.eqiad.wmnet
  • 04:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1034.eqiad.wmnet
  • 04:44 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1034.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 04:40 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2144.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 04:40 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2144.codfw.wmnet
  • 04:40 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2144.codfw.wmnet
  • 04:39 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2143.codfw.wmnet
  • 04:38 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2143.codfw.wmnet
  • 04:38 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2143.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 04:34 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1034.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 04:34 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1034.eqiad.wmnet
  • 04:34 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1034.eqiad.wmnet
  • 04:33 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1021.eqiad.wmnet
  • 04:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1021.eqiad.wmnet
  • 04:31 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1021.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 04:25 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2143.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 04:25 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2143.codfw.wmnet
  • 04:24 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2143.codfw.wmnet
  • 04:24 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2142.codfw.wmnet
  • 04:23 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1021.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 04:23 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1021.eqiad.wmnet
  • 04:22 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1021.eqiad.wmnet
  • 04:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1016.eqiad.wmnet
  • 04:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2142.codfw.wmnet
  • 04:21 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2142.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 04:20 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1016.eqiad.wmnet
  • 04:20 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1016.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 04:11 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2142.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 04:11 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2142.codfw.wmnet
  • 04:10 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2142.codfw.wmnet
  • 04:10 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2141.codfw.wmnet
  • 04:10 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1016.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 04:09 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1016.eqiad.wmnet
  • 04:09 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1016.eqiad.wmnet
  • 04:08 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1015.eqiad.wmnet
  • 04:08 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2141.codfw.wmnet
  • 04:08 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2141.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 04:07 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1015.eqiad.wmnet
  • 04:07 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1015.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 03:57 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2141.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 03:57 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2141.codfw.wmnet
  • 03:57 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1015.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 03:57 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1015.eqiad.wmnet
  • 03:56 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2141.codfw.wmnet
  • 03:56 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D{wikikube-worker[2141-2179].codfw.wmnet} and (A:wikikube-master-codfw or A:wikikube-worker-codfw)
  • 03:56 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1015.eqiad.wmnet
  • 03:56 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D{wikikube-worker[1006-1007,1015-1016,1021,1034-1057].eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad)
  • 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 23s)
  • 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image
  • 01:38 dzahn@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
  • 01:38 dzahn@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
  • 01:38 dzahn@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
  • 01:38 dzahn@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
  • 01:38 dzahn@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
  • 01:37 dzahn@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
  • 00:18 mutante: gerrit - added content-security-policy headers on <Location /r/Documentation/> - style-src 'self' 'unsafe-inline'; font-src 'self' - blocking Google font (T393763)

2026-10-07

  • 22:07 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1152.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 22:05 jclark@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1152.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 22:04 jclark@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-druid1003.eqiad.wmnet with OS bookworm
  • 21:52 bking@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host krb1005.eqiad.wmnet
  • 21:52 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host krb1005.eqiad.wmnet with OS bookworm
  • 21:36 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on krb1005.eqiad.wmnet with reason: host reimage
  • 21:32 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on krb1005.eqiad.wmnet with reason: host reimage
  • 21:22 eevans@deploy1003: helmfile [eqiad] DONE helmfile.d/services/linked-artifacts: apply
  • 21:22 eevans@deploy1003: helmfile [eqiad] START helmfile.d/services/linked-artifacts: apply
  • 21:22 eevans@deploy1003: helmfile [codfw] DONE helmfile.d/services/linked-artifacts: apply
  • 21:21 eevans@deploy1003: helmfile [codfw] START helmfile.d/services/linked-artifacts: apply
  • 21:17 bking@cumin2003: START - Cookbook sre.hosts.reimage for host krb1005.eqiad.wmnet with OS bookworm
  • 21:16 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM krb1005.eqiad.wmnet - bking@cumin2003"
  • 21:16 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM krb1005.eqiad.wmnet - bking@cumin2003"
  • 21:16 bking@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) krb1005.eqiad.wmnet on all recursors
  • 21:16 bking@cumin2003: START - Cookbook sre.dns.wipe-cache krb1005.eqiad.wmnet on all recursors
  • 21:16 bking@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 21:16 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM krb1005.eqiad.wmnet - bking@cumin2003"
  • 21:16 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM krb1005.eqiad.wmnet - bking@cumin2003"
  • 21:08 bking@cumin2003: START - Cookbook sre.dns.netbox
  • 21:08 bking@cumin2003: START - Cookbook sre.ganeti.makevm for new host krb1005.eqiad.wmnet
  • 20:28 kemayo@deploy1003: Finished scap sync-world: Backport for EditSuggestionCounts: Wait for LAC, and retry on timeout (T432733) (duration: 11m 11s)
  • 20:23 kemayo@deploy1003: kemayo: Continuing with deployment
  • 20:21 kemayo@deploy1003: kemayo: Backport for EditSuggestionCounts: Wait for LAC, and retry on timeout (T432733) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 20:17 kemayo@deploy1003: Started scap sync-world: Backport for EditSuggestionCounts: Wait for LAC, and retry on timeout (T432733)
  • 19:55 kemayo@deploy1003: helmfile [codfw] DONE helmfile.d/services/editcheck-headless: apply
  • 19:55 kemayo@deploy1003: helmfile [codfw] START helmfile.d/services/editcheck-headless: apply
  • 19:54 kemayo@deploy1003: helmfile [eqiad] DONE helmfile.d/services/editcheck-headless: apply
  • 19:54 kemayo@deploy1003: helmfile [eqiad] START helmfile.d/services/editcheck-headless: apply
  • 19:19 kemayo@deploy1003: helmfile [staging] DONE helmfile.d/services/editcheck-headless: apply
  • 19:18 kemayo@deploy1003: helmfile [staging] START helmfile.d/services/editcheck-headless: apply
  • 18:33 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.enable_virtualization (exit_code=0) rolling provision on D{wikikube-worker[2124-2140].codfw.wmnet} and (A:wikikube-master-codfw or A:wikikube-worker-codfw)
  • 18:33 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2140.codfw.wmnet
  • 18:33 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2140.codfw.wmnet
  • 18:33 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2140.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 18:22 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2140.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 18:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2140.codfw.wmnet
  • 18:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2140.codfw.wmnet
  • 18:21 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2139.codfw.wmnet
  • 18:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2139.codfw.wmnet
  • 18:21 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2139.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 18:10 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2139.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 18:09 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2139.codfw.wmnet
  • 18:09 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2139.codfw.wmnet
  • 18:09 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2138.codfw.wmnet
  • 18:07 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2138.codfw.wmnet
  • 18:07 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2138.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 17:56 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2138.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 17:56 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2138.codfw.wmnet
  • 17:55 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2138.codfw.wmnet
  • 17:55 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2137.codfw.wmnet
  • 17:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2137.codfw.wmnet
  • 17:53 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2137.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 17:42 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2137.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 17:42 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2137.codfw.wmnet
  • 17:41 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2137.codfw.wmnet
  • 17:41 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2136.codfw.wmnet
  • 17:41 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2136.codfw.wmnet
  • 17:41 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2136.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 17:30 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2136.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 17:30 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2136.codfw.wmnet
  • 17:29 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2136.codfw.wmnet
  • 17:29 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2135.codfw.wmnet
  • 17:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2135.codfw.wmnet
  • 17:27 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2135.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 17:26 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply
  • 17:26 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply
  • 17:25 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply
  • 17:25 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply
  • 17:24 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply
  • 17:24 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply
  • 17:21 dancy@deploy1003: Installation of scap version "4.295.0" completed for 3 hosts
  • 17:19 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply
  • 17:19 dancy@deploy1003: Installing scap version "4.295.0" for 3 host(s)
  • 17:19 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply
  • 17:16 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2135.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 17:16 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2135.codfw.wmnet
  • 17:15 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2135.codfw.wmnet
  • 17:15 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2134.codfw.wmnet
  • 17:15 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2134.codfw.wmnet
  • 17:15 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2134.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 17:12 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply
  • 17:12 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply
  • 17:08 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply
  • 17:08 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply
  • 17:04 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2134.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 17:04 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2134.codfw.wmnet
  • 17:03 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2134.codfw.wmnet
  • 17:03 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2133.codfw.wmnet
  • 17:03 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2133.codfw.wmnet
  • 17:03 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2133.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 16:59 pt1979@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cr1-codfw,cr1-codfw IPv6,re0.cr1-codfw.mgmt
  • 16:59 pt1979@cumin1004: START - Cookbook sre.hosts.remove-downtime for cr1-codfw,cr1-codfw IPv6,re0.cr1-codfw.mgmt
  • 16:57 pt1979@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cr2-codfw,cr2-codfw IPv6,re0.cr2-codfw.mgmt
  • 16:57 pt1979@cumin1004: START - Cookbook sre.hosts.remove-downtime for cr2-codfw,cr2-codfw IPv6,re0.cr2-codfw.mgmt
  • 16:52 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2133.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 16:51 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2133.codfw.wmnet
  • 16:51 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2133.codfw.wmnet
  • 16:50 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2132.codfw.wmnet
  • 16:49 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2132.codfw.wmnet
  • 16:49 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2132.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 16:47 cgoubert@deploy1003: Finished scap sync-world: services_proxy: Enable the linked-artifacts listener for MediaWiki | 1351442 | T432733 (duration: 10m 40s)
  • 16:38 cgoubert@deploy1003: Started scap sync-world: services_proxy: Enable the linked-artifacts listener for MediaWiki | 1351442 | T432733
  • 16:38 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2132.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 16:37 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2132.codfw.wmnet
  • 16:37 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2132.codfw.wmnet
  • 16:37 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2131.codfw.wmnet
  • 16:36 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2131.codfw.wmnet
  • 16:36 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2131.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 16:32 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host testvm2001.codfw.wmnet with OS trixie
  • 16:32 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
  • 16:31 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'.
  • 16:31 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'.
  • 16:29 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'.
  • 16:28 cgoubert@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'.
  • 16:26 cgoubert@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'.
  • 16:26 cgoubert@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
  • 16:25 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2131.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 16:25 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2131.codfw.wmnet
  • 16:24 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2131.codfw.wmnet
  • 16:24 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2130.codfw.wmnet
  • 16:23 cgoubert@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
  • 16:22 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2130.codfw.wmnet
  • 16:22 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply
  • 16:22 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2130.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 16:21 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply
  • 16:21 blake@deploy1003: Finished scap sync-world: add restart policies for T417800 (duration: 05m 23s)
  • 16:18 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on testvm2001.codfw.wmnet with reason: host reimage
  • 16:17 blake@deploy1003: Started scap sync-world: add restart policies for T417800
  • 16:12 elukey@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on testvm2001.codfw.wmnet with reason: host reimage
  • 16:11 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2130.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 16:11 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2130.codfw.wmnet
  • 16:10 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2130.codfw.wmnet
  • 16:10 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host testvm2001.codfw.wmnet (remove config only)
  • 16:10 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2129.codfw.wmnet
  • 16:10 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host testvm2001.codfw.wmnet (remove config only)
  • 16:08 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2129.codfw.wmnet
  • 16:08 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2129.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 16:03 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply
  • 15:59 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply
  • 15:57 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2129.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 15:57 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2129.codfw.wmnet
  • 15:56 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2129.codfw.wmnet
  • 15:56 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2128.codfw.wmnet
  • 15:56 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2128.codfw.wmnet
  • 15:56 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2128.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 15:56 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host testvm2001.codfw.wmnet (push config only)
  • 15:56 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host testvm2001.codfw.wmnet (push config only)
  • 15:55 elukey@cumin1004: START - Cookbook sre.hosts.reimage for host testvm2001.codfw.wmnet with OS trixie
  • 15:51 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr[1-2]-codfw,cr2-codfw IPv6,re0.cr2-codfw.mgmt with reason: cr2-codfw upgrade
  • 15:48 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr1-codfw,cr1-codfw IPv6,re0.cr1-codfw.mgmt with reason: Junos upgrade
  • 15:47 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cr2-codfw,cr2-codfw IPv6,re0.cr2-codfw.mgmt with reason: Junos upgrade
  • 15:45 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2128.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 15:44 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2128.codfw.wmnet
  • 15:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2128.codfw.wmnet
  • 15:44 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2127.codfw.wmnet
  • 15:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2127.codfw.wmnet
  • 15:43 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2127.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 15:43 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2127.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 15:32 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2201.codfw.wmnet with reason: troubleshooting
  • 15:18 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply
  • 15:18 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply
  • 15:11 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr2-eqdfw,cr2-eqdfw IPv6 with reason: Junos upgrade
  • 15:10 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr2-drmrs,cr2-drmrs IPv6,cr2-drmrs.mgmt with reason: Junos upgrade
  • 15:09 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr3-eqsin,cr3-eqsin IPv6,cr3-eqsin.mgmt with reason: Junos upgrade
  • 15:08 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr4-ulsfo,cr4-ulsfo IPv6,cr4-ulsfo.mgmt with reason: Junos upgrade
  • 15:07 pt1979@cumin1004: DONE (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on cr2-ulsfo,cr4-ulsfo IPv6,cr2-ulsfo.mgmt with reason: Junos upgrade
  • 15:07 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr2-esams,cr2-esams IPv6,cr2-esams.mgmt with reason: Junos upgrade
  • 15:07 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr2-eqiad,cr2-eqiad IPv6,re0.cr2-eqiad.mgmt with reason: Junos upgrade
  • 15:06 pt1979@cumin1004: DONE (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on cr2-ceqiad IPv6,cr2-eqiad,re0.cr2-eqiad.mgmt with reason: Junos upgrade
  • 15:06 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on pfw1-codfw with reason: Junos upgrade
  • 15:05 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudsw1-b1-codfw,cloudsw1-b1-codfw IPv6,cloudsw1-b1-codfw.mgmt with reason: Junos upgrade
  • 15:04 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ssw1-f1-codfw,ssw1-f1-codfw IPv6,ssw1-f1-codfw.mgmt with reason: Junos upgrade
  • 15:04 pt1979@cumin1004: DONE (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on ssw1-f8-codfw,ssw1-f1-codfw IPv6,ssw1-f1-codfw.mgmt with reason: Junos upgrade
  • 15:03 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ssw1-d8-codfw,ssw1-d8-codfw IPv6,ssw1-d8-codfw.mgmt with reason: Junos upgrade
  • 15:03 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ssw1-a8-codfw,ssw1-a8-codfw IPv6,ssw1-a8-codfw.mgmt with reason: Junos upgrade
  • 14:53 jforrester@deploy1003: Finished scap sync-world: Backport for WikifunctionsClientUsageUpdateJob: Accept jobs from a later version (T440432) (duration: 10m 09s)
  • 14:48 jforrester@deploy1003: jforrester: Continuing with deployment
  • 14:47 jforrester@deploy1003: jforrester: Backport for WikifunctionsClientUsageUpdateJob: Accept jobs from a later version (T440432) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 14:46 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) upgrade A:liberica-esams
  • 14:44 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.upgrade upgrade A:liberica-esams
  • 14:43 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) upgrade A:liberica-drmrs
  • 14:43 jforrester@deploy1003: Started scap sync-world: Backport for WikifunctionsClientUsageUpdateJob: Accept jobs from a later version (T440432)
  • 14:41 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.upgrade upgrade A:liberica-drmrs
  • 14:41 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) upgrade A:liberica-eqsin
  • 14:40 jforrester@deploy1003: Finished scap sync-world: Backport for Factor out Abstract Wikipedia client mode into a dblist, Wikifunctions: Switch the editor from Ace to CodeMirror (T400015) (duration: 18m 26s)
  • 14:37 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.upgrade upgrade A:liberica-eqsin
  • 14:36 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) upgrade A:liberica-ulsfo
  • 14:34 jforrester@deploy1003: jforrester: Continuing with deployment
  • 14:33 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.upgrade upgrade A:liberica-ulsfo
  • 14:31 oblivian@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-worker2127.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 14:27 moritzm: installing PHP 8.2 security updates
  • 14:27 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) upgrade A:liberica-magru
  • 14:26 jforrester@deploy1003: jforrester: Backport for Factor out Abstract Wikipedia client mode into a dblist, Wikifunctions: Switch the editor from Ace to CodeMirror (T400015) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 14:25 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2127.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 14:24 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2127.codfw.wmnet
  • 14:24 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.upgrade upgrade A:liberica-magru
  • 14:24 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2127.codfw.wmnet
  • 14:24 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2126.codfw.wmnet
  • 14:24 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2126.codfw.wmnet
  • 14:22 jforrester@deploy1003: Started scap sync-world: Backport for Factor out Abstract Wikipedia client mode into a dblist, Wikifunctions: Switch the editor from Ace to CodeMirror (T400015)
  • 14:20 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply
  • 14:20 eevans@deploy1003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply
  • 14:20 aklapper@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.23 refs T438219
  • 14:19 moritzm: installing postgresql-15 security updates
  • 14:18 apine@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
  • 14:18 apine@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
  • 14:18 apine@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
  • 14:17 apine@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
  • 14:16 apine@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
  • 14:16 apine@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
  • 14:14 vgutierrez: upload liberica 0.26 to bookworm-wikimedia (apt.wm.o) - T439840
  • 14:12 apine@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
  • 14:12 apine@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
  • 14:12 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2126.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 14:11 apine@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
  • 14:11 apine@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
  • 14:10 apine@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
  • 14:10 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
  • 14:09 apine@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
  • 14:06 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2126.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 14:06 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2126.codfw.wmnet
  • 14:05 jforrester@deploy1003: Finished scap sync-world: Backport for PublishDialog: Use one slot for the summary field warning (T372149 T440446) (duration: 10m 59s)
  • 14:05 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2126.codfw.wmnet
  • 14:05 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2125.codfw.wmnet
  • 14:05 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2125.codfw.wmnet
  • 14:04 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) restart P{lvs7003.magru.wmnet} and A:liberica
  • 14:04 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) pooling P{lvs7003.magru.wmnet} and A:liberica
  • 14:04 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin pooling P{lvs7003.magru.wmnet} and A:liberica
  • 14:03 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) depooling P{lvs7003.magru.wmnet} and A:liberica
  • 14:03 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin depooling P{lvs7003.magru.wmnet} and A:liberica
  • 14:03 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.upgrade restart P{lvs7003.magru.wmnet} and A:liberica
  • 14:00 jforrester@deploy1003: daphnesmit, jforrester: Continuing with deployment
  • 13:59 jforrester@deploy1003: daphnesmit, jforrester: Backport for PublishDialog: Use one slot for the summary field warning (T372149 T440446) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 13:54 jforrester@deploy1003: Started scap sync-world: Backport for PublishDialog: Use one slot for the summary field warning (T372149 T440446)
  • 13:54 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) restart P{lvs7003.magru.wmnet} and A:liberica
  • 13:54 vgutierrez@cumin1004: END (FAIL) - Cookbook sre.loadbalancer.admin (exit_code=1) pooling P{lvs7003.magru.wmnet} and A:liberica
  • 13:54 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2125.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 13:54 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin pooling P{lvs7003.magru.wmnet} and A:liberica
  • 13:54 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) depooling P{lvs7003.magru.wmnet} and A:liberica
  • 13:53 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin depooling P{lvs7003.magru.wmnet} and A:liberica
  • 13:53 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.upgrade restart P{lvs7003.magru.wmnet} and A:liberica
  • 13:49 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2125.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 13:49 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2125.codfw.wmnet
  • 13:48 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2125.codfw.wmnet
  • 13:48 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2124.codfw.wmnet
  • 13:48 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2124.codfw.wmnet
  • 13:45 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr2-codfw,cr2-codfw IPv6,re0.cr2-codfw.mgmt with reason: Junos upgrade
  • 13:44 papaul: start maintenance on cr2-codfw
  • 13:42 seanleong-wmde@deploy1003: Finished scap sync-world: Backport for Group fix for incorrect group mapping. (T439067) (duration: 15m 31s)
  • 13:37 seanleong-wmde@deploy1003: seanleong-wmde: Continuing with deployment
  • 13:37 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2124.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 13:32 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2124.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 13:31 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2124.codfw.wmnet
  • 13:31 seanleong-wmde@deploy1003: seanleong-wmde: Backport for Group fix for incorrect group mapping. (T439067) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 13:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2124.codfw.wmnet
  • 13:31 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D{wikikube-worker[2124-2140].codfw.wmnet} and (A:wikikube-master-codfw or A:wikikube-worker-codfw)
  • 13:27 cmooney@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device ssw1-a8-eqiad
  • 13:27 seanleong-wmde@deploy1003: Started scap sync-world: Backport for Group fix for incorrect group mapping. (T439067)
  • 13:27 cmooney@cumin1004: START - Cookbook sre.network.tls for network device ssw1-a8-eqiad
  • 13:22 cmooney@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-a8-eqiad
  • 13:21 cmooney@cumin1004: START - Cookbook sre.network.tls for network device lsw1-a8-eqiad
  • 13:19 seanleong-wmde@deploy1003: Finished scap sync-world: Backport for UsageDeduplicator: Re-index C usages after filtering by CQR modifier (T440429) (duration: 11m 07s)
  • 13:15 cmooney@cumin1004: END (FAIL) - Cookbook sre.network.tls (exit_code=99) for network device lsw1-a8-eqiad
  • 13:15 cmooney@cumin1004: START - Cookbook sre.network.tls for network device lsw1-a8-eqiad
  • 13:14 seanleong-wmde@deploy1003: seanleong-wmde: Continuing with deployment
  • 13:13 moritzm: installing mbedtls security updates
  • 13:12 cmooney@cumin1004: END (FAIL) - Cookbook sre.network.tls (exit_code=99) for network device lsw1-a8-eqiad
  • 13:12 seanleong-wmde@deploy1003: seanleong-wmde: Backport for UsageDeduplicator: Re-index C usages after filtering by CQR modifier (T440429) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 13:12 cmooney@cumin1004: START - Cookbook sre.network.tls for network device lsw1-a8-eqiad
  • 13:08 seanleong-wmde@deploy1003: Started scap sync-world: Backport for UsageDeduplicator: Re-index C usages after filtering by CQR modifier (T440429)
  • 13:02 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.enable_virtualization (exit_code=0) rolling provision on D{wikikube-worker[2067-2078,2087-2095,2102-2115].codfw.wmnet} and (A:wikikube-master-codfw or A:wikikube-worker-codfw)
  • 13:02 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2115.codfw.wmnet
  • 13:00 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2115.codfw.wmnet
  • 13:00 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2115.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 12:54 oblivian@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
  • 12:54 oblivian@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
  • 12:54 oblivian@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
  • 12:54 oblivian@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: sync
  • 12:51 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2115.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 12:50 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2115.codfw.wmnet
  • 12:49 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2115.codfw.wmnet
  • 12:49 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2114.codfw.wmnet
  • 12:48 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2114.codfw.wmnet
  • 12:48 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2114.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 12:47 cmooney@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-a8-eqiad
  • 12:47 cmooney@cumin1004: START - Cookbook sre.network.tls for network device lsw1-a8-eqiad
  • 12:47 cmooney@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device ssw1-a8-eqiad
  • 12:46 cmooney@cumin1004: START - Cookbook sre.network.tls for network device ssw1-a8-eqiad
  • 12:45 moritzm: uploaded librsvg 2.60.0+dfsg-1+wmf13u2 to component/thumbor (backport of RUSTSEC-2026-0305 to our internal build)
  • 12:38 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2114.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 12:38 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2114.codfw.wmnet
  • 12:37 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2114.codfw.wmnet
  • 12:37 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2113.codfw.wmnet
  • 12:37 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2113.codfw.wmnet
  • 12:37 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2113.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 12:29 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2113.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 12:29 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2113.codfw.wmnet
  • 12:28 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2113.codfw.wmnet
  • 12:28 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2112.codfw.wmnet
  • 12:28 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2112.codfw.wmnet
  • 12:27 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2112.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 12:26 oblivian@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/jaeger: apply
  • 12:26 oblivian@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/jaeger: apply
  • 12:25 oblivian@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/aux-k8s-services/jaeger: apply
  • 12:25 oblivian@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/aux-k8s-services/jaeger: apply
  • 12:20 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2112.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 12:20 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2112.codfw.wmnet
  • 12:19 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2112.codfw.wmnet
  • 12:19 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2111.codfw.wmnet
  • 12:17 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2111.codfw.wmnet
  • 12:16 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2111.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 12:09 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2111.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 12:09 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2111.codfw.wmnet
  • 12:08 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2111.codfw.wmnet
  • 12:07 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2110.codfw.wmnet
  • 12:05 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2110.codfw.wmnet
  • 12:05 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2110.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 11:57 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2110.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 11:57 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2110.codfw.wmnet
  • 11:56 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2110.codfw.wmnet
  • 11:56 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2109.codfw.wmnet
  • 11:55 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2109.codfw.wmnet
  • 11:54 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2109.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 11:47 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2109.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 11:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2109.codfw.wmnet
  • 11:46 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2109.codfw.wmnet
  • 11:45 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2108.codfw.wmnet
  • 11:45 hnowlan: remove graphite config from puppet, moved graphite hosts to insetup
  • 11:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2108.codfw.wmnet
  • 11:43 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2108.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 11:36 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2108.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 11:36 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2108.codfw.wmnet
  • 11:31 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
  • 11:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2108.codfw.wmnet
  • 11:30 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2107.codfw.wmnet
  • 11:30 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2107.codfw.wmnet
  • 11:30 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2107.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 11:22 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2107.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 11:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2107.codfw.wmnet
  • 11:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2107.codfw.wmnet
  • 11:21 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2106.codfw.wmnet
  • 11:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2106.codfw.wmnet
  • 11:21 aklapper@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.47.0-wmf.23 refs T438219
  • 11:10 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2106.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 11:09 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) restart P{lvs7003.magru.wmnet} and A:liberica
  • 11:09 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) pooling P{lvs7003.magru.wmnet} and A:liberica
  • 11:09 andre: Rolling back again wmf.23 from group1 to group0 - T438219
  • 11:09 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin pooling P{lvs7003.magru.wmnet} and A:liberica
  • 11:09 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) depooling P{lvs7003.magru.wmnet} and A:liberica
  • 11:09 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin depooling P{lvs7003.magru.wmnet} and A:liberica
  • 11:08 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.upgrade restart P{lvs7003.magru.wmnet} and A:liberica
  • 11:05 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2106.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 11:04 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2106.codfw.wmnet
  • 11:04 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2106.codfw.wmnet
  • 11:04 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2105.codfw.wmnet
  • 11:04 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2105.codfw.wmnet
  • 11:02 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P{lvs7003.magru.wmnet} and A:liberica
  • 11:02 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin config_reloading P{lvs7003.magru.wmnet} and A:liberica
  • 10:56 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) restart P{lvs7003.magru.wmnet} and A:liberica
  • 10:55 vgutierrez@cumin1004: END (FAIL) - Cookbook sre.loadbalancer.admin (exit_code=1) pooling P{lvs7003.magru.wmnet} and A:liberica
  • 10:55 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin pooling P{lvs7003.magru.wmnet} and A:liberica
  • 10:55 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) depooling P{lvs7003.magru.wmnet} and A:liberica
  • 10:55 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin depooling P{lvs7003.magru.wmnet} and A:liberica
  • 10:55 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.upgrade restart P{lvs7003.magru.wmnet} and A:liberica
  • 10:50 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2105.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 10:45 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2105.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 10:45 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2105.codfw.wmnet
  • 10:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2105.codfw.wmnet
  • 10:44 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2104.codfw.wmnet
  • 10:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2104.codfw.wmnet
  • 10:39 blake@deploy1003: Finished scap sync-world: update vendored modules for T417800 (duration: 07m 32s)
  • 10:33 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2104.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 10:33 blake@deploy1003: Started scap sync-world: update vendored modules for T417800
  • 10:32 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
  • 10:30 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
  • 10:30 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
  • 10:28 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2104.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 10:27 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2104.codfw.wmnet
  • 10:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2104.codfw.wmnet
  • 10:27 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2103.codfw.wmnet
  • 10:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2103.codfw.wmnet
  • 10:26 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
  • 10:26 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
  • 10:25 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
  • 10:19 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
  • 10:17 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply
  • 10:16 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply
  • 10:16 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
  • 10:14 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2103.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 10:09 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2103.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 10:09 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2103.codfw.wmnet
  • 10:08 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2103.codfw.wmnet
  • 10:07 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2102.codfw.wmnet
  • 10:07 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync
  • 10:07 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync
  • 10:05 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2102.codfw.wmnet
  • 10:05 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2102.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 09:56 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2102.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 09:56 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2102.codfw.wmnet
  • 09:55 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2102.codfw.wmnet
  • 09:55 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2095.codfw.wmnet
  • 09:54 duesen: just a test for SAL
  • 09:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2095.codfw.wmnet
  • 09:53 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2095.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 09:50 daniel@deploy1003: Finished scap sync-world: Backport for REST: Accept message objects as parameter help (T433534 T440377) (duration: 37m 19s)
  • 09:43 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2095.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 09:43 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2095.codfw.wmnet
  • 09:43 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2095.codfw.wmnet
  • 09:42 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2094.codfw.wmnet
  • 09:42 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2094.codfw.wmnet
  • 09:42 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2094.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 09:40 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync
  • 09:39 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync
  • 09:36 daniel@deploy1003: daniel: Continuing with deployment
  • 09:36 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply
  • 09:35 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply
  • 09:32 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2094.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 09:32 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2094.codfw.wmnet
  • 09:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2094.codfw.wmnet
  • 09:31 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2093.codfw.wmnet
  • 09:29 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2093.codfw.wmnet
  • 09:28 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2093.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 09:26 moritzm: restarting pybal on lvs1018 T439730
  • 09:20 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2093.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 09:20 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2093.codfw.wmnet
  • 09:20 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2093.codfw.wmnet
  • 09:19 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2092.codfw.wmnet
  • 09:19 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2092.codfw.wmnet
  • 09:17 daniel@deploy1003: daniel: Backport for REST: Accept message objects as parameter help (T433534 T440377) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 09:13 daniel@deploy1003: Started scap sync-world: Backport for REST: Accept message objects as parameter help (T433534 T440377)
  • 09:02 aqu@deploy1003: Finished deploy [analytics/refinery@43a90b3]: Regular analytics weekly train [analytics/refinery@43a90b33] (duration: 04m 42s)
  • 08:57 aqu@deploy1003: Started deploy [analytics/refinery@43a90b3]: Regular analytics weekly train [analytics/refinery@43a90b33]
  • 08:56 aqu@deploy1003: Finished deploy [analytics/refinery@43a90b3] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@43a90b33] (duration: 00m 44s)
  • 08:55 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2092.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 08:55 aqu@deploy1003: Started deploy [analytics/refinery@43a90b3] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@43a90b33]
  • 08:50 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2092.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 08:50 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2092.codfw.wmnet
  • 08:49 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2092.codfw.wmnet
  • 08:49 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2091.codfw.wmnet
  • 08:49 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2091.codfw.wmnet
  • 08:33 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2091.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 08:29 aklapper@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.23 refs T438219
  • 08:28 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2091.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 08:28 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2091.codfw.wmnet
  • 08:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2091.codfw.wmnet
  • 08:27 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2090.codfw.wmnet
  • 08:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2090.codfw.wmnet
  • 08:20 aklapper@deploy1003: Rolling back deployment
  • 08:14 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2090.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 08:09 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2090.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 08:08 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2090.codfw.wmnet
  • 08:08 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2090.codfw.wmnet
  • 08:07 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2089.codfw.wmnet
  • 08:07 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2089.codfw.wmnet
  • 08:02 awight@deploy1003: Finished scap sync-world: Backport for Adds metrics for entity usage limit baseline (T439125) (duration: 37m 41s)
  • 07:57 awight@deploy1003: nicholusmuwonge, awight: Continuing with deployment
  • 07:56 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2089.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 07:51 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2089.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 07:51 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2089.codfw.wmnet
  • 07:50 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2089.codfw.wmnet
  • 07:50 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2088.codfw.wmnet
  • 07:50 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2088.codfw.wmnet
  • 07:42 awight@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply
  • 07:41 awight@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply
  • 07:40 elukey@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'sync'.
  • 07:40 elukey@deploy1003: helmfile [eqiad] START helmfile.d/admin 'sync'.
  • 07:40 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2088.codfw.wmnet
  • 07:39 elukey@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'sync'.
  • 07:39 elukey@deploy1003: helmfile [codfw] START helmfile.d/admin 'sync'.
  • 07:39 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2088.codfw.wmnet
  • 07:39 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2087.codfw.wmnet
  • 07:39 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2087.codfw.wmnet
  • 07:39 elukey@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'.
  • 07:39 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2087.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 07:39 elukey@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'.
  • 07:38 elukey@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'.
  • 07:38 elukey@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'.
  • 07:37 elukey@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'sync'.
  • 07:37 elukey@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'sync'.
  • 07:36 elukey@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'sync'.
  • 07:36 elukey@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'sync'.
  • 07:35 elukey@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'.
  • 07:35 elukey@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'.
  • 07:34 elukey@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'sync'.
  • 07:34 elukey@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'sync'.
  • 07:33 elukey@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'.
  • 07:33 elukey@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'.
  • 07:30 moritzm: uploaded openjdk-21 21.0.12.1+1-1~deb12u1 to component/jdk21 for bookworm-wikimedia T392465
  • 07:29 awight@deploy1003: nicholusmuwonge, awight: Backport for Adds metrics for entity usage limit baseline (T439125) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 07:28 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2087.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 07:28 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2087.codfw.wmnet
  • 07:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2087.codfw.wmnet
  • 07:27 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2078.codfw.wmnet
  • 07:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2078.codfw.wmnet
  • 07:27 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2078.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 07:26 elukey@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'sync'.
  • 07:26 elukey@deploy1003: helmfile [codfw] START helmfile.d/admin 'sync'.
  • 07:25 elukey@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'sync'.
  • 07:25 awight@deploy1003: Started scap sync-world: Backport for Adds metrics for entity usage limit baseline (T439125)
  • 07:24 elukey@deploy1003: helmfile [eqiad] START helmfile.d/admin 'sync'.
  • 07:22 elukey@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'.
  • 07:22 elukey@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'.
  • 07:20 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2245: Pooling back after cloning db2250
  • 07:19 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2078.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 07:19 elukey@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'.
  • 07:19 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2078.codfw.wmnet
  • 07:18 elukey@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'.
  • 07:18 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2078.codfw.wmnet
  • 07:18 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2077.codfw.wmnet
  • 07:17 elukey@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'.
  • 07:16 elukey@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'.
  • 07:16 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2077.codfw.wmnet
  • 07:16 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2077.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 07:14 elukey@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'.
  • 07:14 elukey@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'.
  • 07:12 elukey@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'sync'.
  • 07:11 elukey@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'sync'.
  • 07:08 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2077.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 07:08 elukey@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'sync'.
  • 07:08 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2077.codfw.wmnet
  • 07:08 elukey@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'sync'.
  • 07:07 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2077.codfw.wmnet
  • 07:06 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2076.codfw.wmnet
  • 07:06 elukey@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'sync'.
  • 07:05 elukey@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'sync'.
  • 07:04 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2076.codfw.wmnet
  • 07:04 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2076.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 06:56 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2076.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 06:56 moritzm: installing bird security updates
  • 06:56 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2076.codfw.wmnet
  • 06:55 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2076.codfw.wmnet
  • 06:55 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2075.codfw.wmnet
  • 06:55 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2075.codfw.wmnet
  • 06:44 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2075.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 06:39 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2075.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 06:39 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2075.codfw.wmnet
  • 06:38 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2075.codfw.wmnet
  • 06:38 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2074.codfw.wmnet
  • 06:38 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2074.codfw.wmnet
  • 06:35 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db2245: Pooling back after cloning db2250
  • 06:27 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2074.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 06:22 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2074.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 06:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2074.codfw.wmnet
  • 06:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2074.codfw.wmnet
  • 06:21 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2073.codfw.wmnet
  • 06:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2073.codfw.wmnet
  • 06:20 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2073.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 06:11 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2073.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 06:11 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2073.codfw.wmnet
  • 06:10 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2073.codfw.wmnet
  • 06:10 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2072.codfw.wmnet
  • 06:10 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2072.codfw.wmnet
  • 06:09 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2072.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 06:01 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2072.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 06:01 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2072.codfw.wmnet
  • 06:00 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2072.codfw.wmnet
  • 06:00 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2071.codfw.wmnet
  • 05:59 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2071.codfw.wmnet
  • 05:59 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2071.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 05:54 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2250.codfw.wmnet with reason: cloning
  • 05:51 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2071.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 05:50 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2071.codfw.wmnet
  • 05:50 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2071.codfw.wmnet
  • 05:50 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2070.codfw.wmnet
  • 05:48 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2070.codfw.wmnet
  • 05:48 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2070.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 05:41 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2250.codfw.wmnet with reason: cloning
  • 05:40 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2070.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 05:40 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2070.codfw.wmnet
  • 05:40 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2070.codfw.wmnet
  • 05:39 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2069.codfw.wmnet
  • 05:39 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2069.codfw.wmnet
  • 05:39 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2069.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 05:32 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2069.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 05:31 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2069.codfw.wmnet
  • 05:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2069.codfw.wmnet
  • 05:31 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2068.codfw.wmnet
  • 05:30 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2068.codfw.wmnet
  • 05:29 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2068.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 05:21 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2068.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 05:21 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2068.codfw.wmnet
  • 05:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2068.codfw.wmnet
  • 05:20 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2067.codfw.wmnet
  • 05:18 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2067.codfw.wmnet
  • 05:18 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2067.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 05:10 kart_: Updated Apertium to 2026-09-30-051826-production (T437213)
  • 05:09 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2067.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 05:09 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2067.codfw.wmnet
  • 05:08 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2067.codfw.wmnet
  • 05:08 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D{wikikube-worker[2067-2078,2087-2095,2102-2115].codfw.wmnet} and (A:wikikube-master-codfw or A:wikikube-worker-codfw)
  • 05:04 kartik@deploy1003: helmfile [eqiad] DONE helmfile.d/services/apertium: apply
  • 05:04 kartik@deploy1003: helmfile [eqiad] START helmfile.d/services/apertium: apply
  • 05:01 kartik@deploy1003: helmfile [codfw] DONE helmfile.d/services/apertium: apply
  • 05:00 kartik@deploy1003: helmfile [codfw] START helmfile.d/services/apertium: apply
  • 04:54 marostegui: Stop mariadb on x4 db2245 to clone the backup host T437279
  • 04:52 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2245.codfw.wmnet with reason: cloning
  • 04:51 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2245: to clone the backup host T437279
  • 04:51 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool db2245: to clone the backup host T437279
  • 04:50 marostegui: Stop mariadb on x4 db2245 to clone the backup host
  • 04:50 kartik@deploy1003: helmfile [staging] DONE helmfile.d/services/apertium: apply
  • 04:50 kartik@deploy1003: helmfile [staging] START helmfile.d/services/apertium: apply
  • 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 35s)
  • 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image

2026-10-06

  • 22:07 jdlrobson@deploy1003: Finished scap sync-world: Backport for Revert "Scroll the TOC along with the page scrolling" (T440296) (duration: 11m 24s)
  • 22:01 jdlrobson@deploy1003: jdlrobson: Continuing with deployment
  • 22:00 jdlrobson@deploy1003: jdlrobson: Backport for Revert "Scroll the TOC along with the page scrolling" (T440296) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 21:55 jdlrobson@deploy1003: Started scap sync-world: Backport for Revert "Scroll the TOC along with the page scrolling" (T440296)
  • 20:15 catrope@deploy1003: Finished scap sync-world: Backport for Email confirmation: Temporarily disable banner modification, Email confirmation: Temporarily disable banner modification, EmailConfirmationHooks: Replace hard-coded timestamp with getStartDate(), EmailConfirmationHooks: Replace hard-coded timestamp with getStartDate() (duration: 12m 17
  • 20:10 catrope@deploy1003: catrope: Continuing with deployment
  • 20:07 catrope@deploy1003: catrope: Backport for Email confirmation: Temporarily disable banner modification, Email confirmation: Temporarily disable banner modification, EmailConfirmationHooks: Replace hard-coded timestamp with getStartDate(), EmailConfirmationHooks: Replace hard-coded timestamp with getStartDate() synced to the testservers (see htt
  • 20:03 catrope@deploy1003: Started scap sync-world: Backport for Email confirmation: Temporarily disable banner modification, Email confirmation: Temporarily disable banner modification, EmailConfirmationHooks: Replace hard-coded timestamp with getStartDate(), EmailConfirmationHooks: Replace hard-coded timestamp with getStartDate()
  • 18:48 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply
  • 18:48 eevans@deploy1003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply
  • 18:32 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1218: Pooling back db1218 after stress test
  • 18:32 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2209: Pooling back db2209 after stress test
  • 18:32 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2207: Pooling back db2207 after stress test
  • 18:02 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2055-2062,2064-2065].codfw.wmnet
  • 18:01 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2055-2062,2064-2065].codfw.wmnet
  • 17:47 marostegui: Repool db1218, db2207 and db2209 after stress testing
  • 17:47 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db1218: Pooling back db1218 after stress test
  • 17:47 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db2209: Pooling back db2209 after stress test
  • 17:47 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db2207: Pooling back db2207 after stress test
  • 17:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.enable_virtualization (exit_code=0) rolling provision on D{wikikube-worker[2055-2062,2064-2065].codfw.wmnet} and (A:wikikube-master-codfw or A:wikikube-worker-codfw)
  • 17:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2065.codfw.wmnet
  • 17:46 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2065.codfw.wmnet
  • 17:46 marostegui@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2207: Pooling back db2207 after stress test
  • 17:45 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2065.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 17:44 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db2207: Pooling back db2207 after stress test
  • 17:36 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2065.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 17:36 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2065.codfw.wmnet
  • 17:36 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2065.codfw.wmnet
  • 17:35 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2064.codfw.wmnet
  • 17:34 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2064.codfw.wmnet
  • 17:34 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2064.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 17:33 fceratto@cumin1004: END (FAIL) - Cookbook sre.mysql.pool (exit_code=99) pool db1901: test
  • 17:25 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2064.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 17:24 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2064.codfw.wmnet
  • 17:24 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2064.codfw.wmnet
  • 17:24 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2062.codfw.wmnet
  • 17:23 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2062.codfw.wmnet
  • 17:22 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2062.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 17:15 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2062.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 17:14 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2062.codfw.wmnet
  • 17:14 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2062.codfw.wmnet
  • 17:13 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2061.codfw.wmnet
  • 17:13 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2061.codfw.wmnet
  • 17:12 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2061.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 17:12 bking@cumin2003: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM matomo1003.eqiad.wmnet
  • 17:08 bking@cumin2003: START - Cookbook sre.ganeti.reboot-vm for VM matomo1003.eqiad.wmnet
  • 17:05 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2061.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 17:05 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2061.codfw.wmnet
  • 17:04 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2061.codfw.wmnet
  • 17:04 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2060.codfw.wmnet
  • 17:04 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2060.codfw.wmnet
  • 17:03 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2060.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 16:53 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2060.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 16:53 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2060.codfw.wmnet
  • 16:52 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2060.codfw.wmnet
  • 16:52 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2059.codfw.wmnet
  • 16:51 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2059.codfw.wmnet
  • 16:51 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2059.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 16:43 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2059.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 16:43 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2059.codfw.wmnet
  • 16:42 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2059.codfw.wmnet
  • 16:42 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2058.codfw.wmnet
  • 16:42 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2058.codfw.wmnet
  • 16:41 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2058.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 16:39 bking@dns2004: END - running authdns-update
  • 16:37 bking@dns2004: START - running authdns-update
  • 16:34 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2058.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 16:33 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2058.codfw.wmnet
  • 16:33 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2058.codfw.wmnet
  • 16:32 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2057.codfw.wmnet
  • 16:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2057.codfw.wmnet
  • 16:31 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2057.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 16:23 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2057.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 16:23 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2057.codfw.wmnet
  • 16:22 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2057.codfw.wmnet
  • 16:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2056.codfw.wmnet
  • 16:22 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2056.codfw.wmnet
  • 16:21 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2056.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 16:14 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2056.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 16:13 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2056.codfw.wmnet
  • 16:13 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2056.codfw.wmnet
  • 16:12 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2055.codfw.wmnet
  • 16:12 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2055.codfw.wmnet
  • 16:12 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2055.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 16:05 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2055.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 16:04 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2055.codfw.wmnet
  • 16:04 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2055.codfw.wmnet
  • 16:04 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D{wikikube-worker[2055-2062,2064-2065].codfw.wmnet} and (A:wikikube-master-codfw or A:wikikube-worker-codfw)
  • 15:58 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-mcrouter: apply
  • 15:56 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-mcrouter: apply
  • 15:56 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-mcrouter: apply
  • 15:56 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply
  • 15:55 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-mcrouter: apply
  • 15:55 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-mcrouter: apply
  • 15:54 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-mcrouter: apply
  • 15:46 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply
  • 15:45 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
  • 15:45 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
  • 15:36 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
  • 15:35 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
  • 15:26 daniel@deploy1003: Finished scap sync-world: Backport for REST: enable the content/v2-beta module on testwiki (T433534) (duration: 15m 44s)
  • 15:21 daniel@deploy1003: daniel: Continuing with deployment
  • 15:17 daniel@deploy1003: daniel: Backport for REST: enable the content/v2-beta module on testwiki (T433534) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 15:11 moritzm: installing python3.13 security updates
  • 15:11 jayme@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
  • 15:11 jayme@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'.
  • 15:11 daniel@deploy1003: Started scap sync-world: Backport for REST: enable the content/v2-beta module on testwiki (T433534)
  • 15:11 jayme@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'.
  • 15:10 jayme@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'.
  • 15:08 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
  • 15:07 cdobbins@cumin1004: END (PASS) - Cookbook sre.cdn.roll-restart-ats (exit_code=0) rolling restart_daemons on A:cp-codfw
  • 15:06 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
  • 15:04 brennen@deploy1003: Finished deploy [phabricator/deployment@2b620eb]: deploy phab1005 for T440340 (duration: 00m 42s)
  • 15:03 brennen@deploy1003: Started deploy [phabricator/deployment@2b620eb]: deploy phab1005 for T440340
  • 15:03 aokoth@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on phab1005.eqiad.wmnet with reason: Deployment
  • 15:03 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
  • 15:03 jayme@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
  • 15:03 jayme@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
  • 15:03 brennen@deploy1003: Finished deploy [phabricator/deployment@2b620eb]: deploy phab2003 for T440340 (duration: 00m 51s)
  • 15:02 jayme@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'.
  • 15:02 jayme@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'.
  • 15:02 brennen@deploy1003: Started deploy [phabricator/deployment@2b620eb]: deploy phab2003 for T440340
  • 15:02 aokoth@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on phab2003.codfw.wmnet with reason: Deployment
  • 15:02 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply
  • 14:59 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
  • 14:59 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply
  • 14:57 moritzm: restarting pybal on lvs1018 T439730
  • 14:51 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2041-2042,2044,2046,2049-2051].codfw.wmnet
  • 14:51 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2041-2042,2044,2046,2049-2051].codfw.wmnet
  • 14:48 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.enable_virtualization (exit_code=0) rolling provision on D{wikikube-worker[2041-2042,2044,2046,2049-2051].codfw.wmnet} and (A:wikikube-master-codfw or A:wikikube-worker-codfw)
  • 14:48 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2051.codfw.wmnet
  • 14:48 moritzm: restarting pybal on lvs1020 T439730
  • 14:47 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2051.codfw.wmnet
  • 14:47 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2051.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 14:41 cdobbins@cumin1004: START - Cookbook sre.cdn.roll-restart-ats rolling restart_daemons on A:cp-codfw
  • 14:40 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2051.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 14:40 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2051.codfw.wmnet
  • 14:39 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2051.codfw.wmnet
  • 14:39 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2050.codfw.wmnet
  • 14:38 ladsgroup@deploy1003: Finished scap sync-world: Backport for Enable $wgCiteDefaultRefDirAuto on fawiki (T432544) (duration: 11m 37s)
  • 14:38 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2050.codfw.wmnet
  • 14:38 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2050.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 14:33 ladsgroup@deploy1003: ladsgroup, mamouri: Continuing with deployment
  • 14:33 moritzm: restarting pybal on lvs2012 T439730
  • 14:32 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2050.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 14:32 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2050.codfw.wmnet
  • 14:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2050.codfw.wmnet
  • 14:31 cdobbins@cumin1004: END (PASS) - Cookbook sre.cdn.roll-restart-ats (exit_code=0) rolling restart_daemons on A:cp-eqsin
  • 14:31 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2049.codfw.wmnet
  • 14:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2049.codfw.wmnet
  • 14:31 ladsgroup@deploy1003: ladsgroup, mamouri: Backport for Enable $wgCiteDefaultRefDirAuto on fawiki (T432544) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 14:31 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2049.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 14:28 trueg@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply
  • 14:27 ladsgroup@deploy1003: Started scap sync-world: Backport for Enable $wgCiteDefaultRefDirAuto on fawiki (T432544)
  • 14:27 trueg@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply
  • 14:23 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2049.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 14:23 jelto@deploy1003: Finished scap deploy-service miscweb: First test deployment of miscweb using deploy-service and spiderpig (duration: 01m 54s)
  • 14:23 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2049.codfw.wmnet
  • 14:23 moritzm: restarting pybal on lvs2014 T439730
  • 14:23 jelto@deploy1003: scap deploy-service miscweb: [PRODUCTION/DEFAULT] eqiad Deploying miscweb
  • 14:22 jelto@deploy1003: scap deploy-service miscweb: [PRODUCTION/DEFAULT] codfw Deploying miscweb
  • 14:22 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2049.codfw.wmnet
  • 14:22 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2046.codfw.wmnet
  • 14:21 jelto@deploy1003: scap deploy-service miscweb: [STAGING/DEFAULT] staging-eqiad Deploying miscweb
  • 14:21 jelto@deploy1003: Started scap deploy-service miscweb: First test deployment of miscweb using deploy-service and spiderpig
  • 14:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2046.codfw.wmnet
  • 14:20 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2046.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 14:13 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2046.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 14:13 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2046.codfw.wmnet
  • 14:12 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2046.codfw.wmnet
  • 14:12 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2044.codfw.wmnet
  • 14:12 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2044.codfw.wmnet
  • 14:11 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2044.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • {{safesubst:SAL entry|1=14:08 ladsgroup@deploy1003: Finished scap sync-world: Backport for fix(AccountSetup): Send onboarding exposure from the post-signup redirect, refactor(AccountSetup): Move post-signup onboarding checks to a shared service, fix(AccountSetup): Send onboarding exposure from the post-signup redirect, [[gerrit:1352031|refactor(AccountSetup): Move post-signup onboarding chec}}
  • 14:06 ladsgroup@deploy1003: migr, ladsgroup: Continuing with deployment
  • 14:06 cdobbins@cumin1004: START - Cookbook sre.cdn.roll-restart-ats rolling restart_daemons on A:cp-eqsin
  • 14:04 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2044.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 14:04 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2044.codfw.wmnet
  • 14:03 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2044.codfw.wmnet
  • 14:03 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2042.codfw.wmnet
  • {{safesubst:SAL entry|1=14:03 ladsgroup@deploy1003: migr, ladsgroup: Backport for fix(AccountSetup): Send onboarding exposure from the post-signup redirect, refactor(AccountSetup): Move post-signup onboarding checks to a shared service, fix(AccountSetup): Send onboarding exposure from the post-signup redirect, [[gerrit:1352031|refactor(AccountSetup): Move post-signup onboarding checks to a s}}
  • 14:02 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2042.codfw.wmnet
  • 14:02 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2042.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • {{safesubst:SAL entry|1=14:01 ladsgroup@deploy1003: Started scap sync-world: Backport for fix(AccountSetup): Send onboarding exposure from the post-signup redirect, refactor(AccountSetup): Move post-signup onboarding checks to a shared service, fix(AccountSetup): Send onboarding exposure from the post-signup redirect, [[gerrit:1352031|refactor(AccountSetup): Move post-signup onboarding check}}
  • 13:55 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply
  • 13:55 ladsgroup@deploy1003: Finished scap sync-world: Backport for ltwiki: Enable extended confirmed user group (T440216) (duration: 13m 47s)
  • 13:55 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply
  • 13:55 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply
  • 13:55 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply
  • 13:54 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2042.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 13:54 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2042.codfw.wmnet
  • 13:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2042.codfw.wmnet
  • 13:53 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D{wikikube-worker[2041-2042,2044,2046,2049-2051].codfw.wmnet} and (A:wikikube-master-codfw or A:wikikube-worker-codfw)
  • 13:50 ladsgroup@deploy1003: tryvix1509, ladsgroup: Continuing with deployment
  • 13:46 ladsgroup@deploy1003: tryvix1509, ladsgroup: Backport for ltwiki: Enable extended confirmed user group (T440216) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 13:42 ladsgroup@deploy1003: Started scap sync-world: Backport for ltwiki: Enable extended confirmed user group (T440216)
  • 13:38 oblivian@cumin1004: END (ERROR) - Cookbook sre.k8s.enable_virtualization (exit_code=97) rolling provision on D{wikikube-worker[2041-2042,2044,2046,2049-2051].codfw.wmnet} and (A:wikikube-master-codfw or A:wikikube-worker-codfw)
  • 13:38 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2041.codfw.wmnet
  • 13:38 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2041.codfw.wmnet
  • 13:38 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2041.codfw.wmnet
  • 13:37 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2041.codfw.wmnet
  • 13:37 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D{wikikube-worker[2041-2042,2044,2046,2049-2051].codfw.wmnet} and (A:wikikube-master-codfw or A:wikikube-worker-codfw)
  • 13:35 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.enable_virtualization (exit_code=0) rolling provision on D{wikikube-worker[2033-2039].codfw.wmnet} and (A:wikikube-master-codfw or A:wikikube-worker-codfw)
  • 13:35 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D{wikikube-worker[2033-2039].codfw.wmnet} and (A:wikikube-master-codfw or A:wikikube-worker-codfw)
  • 13:34 moritzm: installing php8.2 security updates
  • 13:34 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.enable_virtualization (exit_code=0) rolling provision on D{wikikube-worker[2011-2018].codfw.wmnet} and (A:wikikube-master-codfw or A:wikikube-worker-codfw)
  • 13:34 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D{wikikube-worker[2011-2018].codfw.wmnet} and (A:wikikube-master-codfw or A:wikikube-worker-codfw)
  • 13:32 elukey@deploy1003: Unlocked for deployment [MediaWiki]: Drop old MediaWiki images from the Registry - T437297 (duration: 214m 10s)
  • 13:31 slyngshede@dns1004: END - running authdns-update
  • 13:29 slyngshede@dns1004: START - running authdns-update
  • 13:25 moritzm: installing libssh2 security updates
  • 13:20 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
  • 13:18 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
  • 13:14 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
  • 13:14 moritzm: installing glibc security updates
  • 13:11 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply
  • 13:04 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
  • 13:04 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply
  • 13:03 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/proton: apply
  • 13:02 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/proton: apply
  • 12:59 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/proton: apply
  • 12:58 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/proton: apply
  • 12:45 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/proton: apply
  • 12:44 jmm@deploy1003: helmfile [staging] START helmfile.d/services/proton: apply
  • 12:41 ihurbain@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply
  • 12:40 ihurbain@deploy1003: helmfile [eqiad] START helmfile.d/services/mobileapps: apply
  • 12:40 ihurbain@deploy1003: helmfile [codfw] DONE helmfile.d/services/mobileapps: apply
  • 12:39 ihurbain@deploy1003: helmfile [codfw] START helmfile.d/services/mobileapps: apply
  • 12:39 ihurbain@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
  • 12:38 Amir1: sudo cumin --force --no-color "A:db-all" 'find /srv/sqldata*/db2*-bin.* -maxdepth 1 -type f -mtime +365 -delete'
  • 12:38 ihurbain@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
  • 12:38 Amir1: sudo cumin --force --no-color "A:db-all" 'find /srv/sqldata*/db1*-bin.* -maxdepth 1 -type f -mtime +365 -delete'
  • 12:03 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync
  • 12:02 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync
  • 11:56 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync
  • 11:56 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync
  • 11:52 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
  • 11:41 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-mcrouter: apply
  • 11:41 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-mcrouter: apply
  • 11:33 moritzm: installing chromium security updates
  • 11:21 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephosd1056.eqiad.wmnet with OS bookworm
  • 11:21 filippo@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - filippo@cumin1004"
  • 11:21 filippo@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - filippo@cumin1004"
  • 11:09 blake@deploy1003: helmfile [staging] DONE helmfile.d/services/mw-mcrouter: apply
  • 11:08 blake@deploy1003: helmfile [staging] START helmfile.d/services/mw-mcrouter: apply
  • 11:05 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 11:02 jmm@cumin2003: START - Cookbook sre.dns.netbox
  • 11:01 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephosd1056.eqiad.wmnet with reason: host reimage
  • 10:56 filippo@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephosd1056.eqiad.wmnet with reason: host reimage
  • 10:40 filippo@cumin1004: START - Cookbook sre.hosts.reimage for host cloudcephosd1056.eqiad.wmnet with OS bookworm
  • 10:38 filippo@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudcephosd1056.eqiad.wmnet with OS bookworm
  • 10:38 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephosd1055.eqiad.wmnet with OS bookworm
  • 10:38 filippo@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - filippo@cumin1004"
  • 10:38 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2228: pooling in s7
  • 10:37 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2169: Pooling back
  • 10:37 filippo@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - filippo@cumin1004"
  • 10:19 filippo@cumin1004: START - Cookbook sre.hosts.reimage for host cloudcephosd1056.eqiad.wmnet with OS bookworm
  • 10:18 filippo@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudcephosd1056.eqiad.wmnet with OS bookworm
  • 10:18 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephosd1055.eqiad.wmnet with reason: host reimage
  • 10:14 Amir1: mwscript-k8s --dblist=all -- purgeUserOptions.php --login-age 5 visualeditor-editor (T406724)
  • 10:13 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool pc1021: Pooling back pc1
  • 10:13 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0)
  • 10:13 marostegui@cumin1004: START - Cookbook sre.mysql.parsercache
  • 10:13 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool pc1021: Pooling back pc1
  • 10:12 filippo@cumin1004: START - Cookbook sre.hosts.reimage for host cloudcephosd1056.eqiad.wmnet with OS bookworm
  • 10:12 filippo@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephosd1055.eqiad.wmnet with reason: host reimage
  • 10:10 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-mcrouter: apply
  • 10:10 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-mcrouter: apply
  • 10:10 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-mcrouter: apply
  • 10:10 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:15:00 on pc2021.codfw.wmnet,pc1021.eqiad.wmnet with reason: upgrading
  • 10:10 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-mcrouter: apply
  • 10:09 gkyziridis@deploy1003: helmfile [ml-serve-codfw] 'sync' command on namespace 'liftwing-openapi-server' for release 'main' .
  • 10:09 marostegui: Upgrade pc1 to mariadb 10.11.19 T436082
  • 10:09 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'liftwing-openapi-server' for release 'main' .
  • 10:09 moritzm: installing curl security updates
  • 10:08 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool pc1021: testing
  • 10:08 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0)
  • 10:08 marostegui@cumin1004: START - Cookbook sre.mysql.parsercache
  • 10:08 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool pc1021: testing
  • 10:02 elukey: `elukey@registry1004:~$ time sudo -u docker-registry /usr/bin/docker-registry garbage-collect /etc/docker/registry/config-restricted.yml -m | tee gc_run_06102026` - T437297
  • 09:58 gkyziridis@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revision-models' for release 'main' .
  • 09:58 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revision-models' for release 'main' .
  • 09:58 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2168: Pooling in s7
  • 09:58 elukey@deploy1003: Locking from deployment [MediaWiki]: Drop old MediaWiki images from the Registry - T437297
  • 09:57 elukey@deploy1003: Unlocked for deployment [MediaWiki]: Drop old MediaWiki images from the Registry - T437297 (duration: 00m 19s)
  • 09:57 elukey@deploy1003: Locking from deployment [MediaWiki]: Drop old MediaWiki images from the Registry - T437297
  • 09:55 filippo@cumin1004: START - Cookbook sre.hosts.reimage for host cloudcephosd1055.eqiad.wmnet with OS bookworm
  • 09:53 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db2228: pooling in s7
  • 09:52 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db2169: Pooling back
  • 09:45 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-mcrouter: apply
  • 09:45 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-mcrouter: apply
  • 09:45 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-mcrouter: apply
  • 09:45 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-mcrouter: apply
  • 09:32 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
  • 09:15 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2012.codfw.wmnet
  • 09:15 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2012.codfw.wmnet
  • 09:15 oblivian@cumin1004: END (ERROR) - Cookbook sre.k8s.enable_virtualization (exit_code=97) rolling provision on D{wikikube-worker[2011-2018].codfw.wmnet} and (A:wikikube-master-codfw or A:wikikube-worker-codfw)
  • 09:14 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2011.codfw.wmnet
  • 09:14 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2011.codfw.wmnet
  • 09:14 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2011.codfw.wmnet
  • 09:13 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2011.codfw.wmnet
  • 09:13 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D{wikikube-worker[2011-2018].codfw.wmnet} and (A:wikikube-master-codfw or A:wikikube-worker-codfw)
  • 09:13 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db2168: Pooling in s7
  • 09:13 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2168: testing
  • 09:12 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool db2168: testing
  • 09:12 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1901: test
  • 09:11 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
  • 09:02 dkertesz@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-esams and A:cp - upgrade haproxy to 3.2.25-1 (T439667)
  • 09:00 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync
  • 09:00 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync
  • 08:56 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
  • 08:54 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
  • 08:30 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db[2169,2228].codfw.wmnet with reason: cloning
  • 08:30 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2228: moving cloning
  • 08:29 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool db2228: moving cloning
  • 08:29 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2169: moving cloning
  • 08:29 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool db2169: moving cloning
  • 08:28 fceratto@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1901: test
  • 08:28 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1901: test
  • 08:28 aklapper@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.47.0-wmf.23 refs T438219
  • 08:17 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
  • 08:16 fceratto@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1901: test
  • 08:16 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1901: test
  • 08:14 fceratto@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1901: test
  • 08:14 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1901: test
  • 08:14 fceratto@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1901: test
  • 08:14 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1901: test
  • 08:14 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
  • 08:14 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
  • 08:14 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
  • 08:14 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: sync
  • 08:13 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
  • 08:10 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2169: Pooling in s7
  • 08:04 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2168: Pooling back
  • 08:00 matthiasmullie: UTC morning backport window done
  • 08:00 mlitn@deploy1003: Finished scap sync-world: Backport for Properly render carousel preview dialog sizing in Webkit (T439007), Change lead section infobox exclusion (T439943), [Image Browsing] Prevent layout shift when loading carousel experiment (T439948) (duration: 10m 18s)
  • 07:55 mlitn@deploy1003: mlitn, ksarabia: Continuing with deployment
  • 07:54 mlitn@deploy1003: mlitn, ksarabia: Backport for Properly render carousel preview dialog sizing in Webkit (T439007), Change lead section infobox exclusion (T439943), [Image Browsing] Prevent layout shift when loading carousel experiment (T439948) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 07:50 mlitn@deploy1003: Started scap sync-world: Backport for Properly render carousel preview dialog sizing in Webkit (T439007), Change lead section infobox exclusion (T439943), [Image Browsing] Prevent layout shift when loading carousel experiment (T439948)
  • 07:45 dcausse@deploy1003: Finished scap sync-world: Backport for Do not suggest duplicates when completing special pages (T440147), Find canonical pages on special page completion search (T440151) (duration: 13m 53s)
  • 07:38 dcausse@deploy1003: dcausse: Continuing with deployment
  • 07:38 dcausse@deploy1003: dcausse: Backport for Do not suggest duplicates when completing special pages (T440147), Find canonical pages on special page completion search (T440151) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 07:34 fceratto@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1901: test
  • 07:33 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1901: test
  • 07:31 dcausse@deploy1003: Started scap sync-world: Backport for Do not suggest duplicates when completing special pages (T440147), Find canonical pages on special page completion search (T440151)
  • 07:30 dkertesz@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-esams and A:cp - upgrade haproxy to 3.2.25-1 (T439667)
  • 07:25 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db2169: Pooling in s7
  • 07:21 awight@deploy1003: Finished scap sync-world: Backport for Config change for launch of stopping sending LL notifications into group1. (T437577) (duration: 17m 31s)
  • 07:15 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db2168: Pooling back
  • 07:08 awight@deploy1003: awight, seanleong-wmde: Continuing with deployment
  • 07:08 awight@deploy1003: awight, seanleong-wmde: Backport for Config change for launch of stopping sending LL notifications into group1. (T437577) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 07:03 awight@deploy1003: Started scap sync-world: Backport for Config change for launch of stopping sending LL notifications into group1. (T437577)
  • 05:36 marostegui@cumin1004: END (FAIL) - Cookbook sre.mysql.clone (exit_code=99) of db2168.codfw.wmnet onto db2169.codfw.wmnet
  • 05:36 marostegui@cumin1004: START - Cookbook sre.mysql.clone of db2168.codfw.wmnet onto db2169.codfw.wmnet
  • 05:31 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2168: moving to s7
  • 05:31 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool db2168: moving to s7
  • 05:28 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db[2168-2169].codfw.wmnet with reason: cloning
  • 05:26 marostegui@cumin1004: END (FAIL) - Cookbook sre.mysql.clone (exit_code=99) of db2168.codfw.wmnet onto db2169.codfw.wmnet
  • 05:26 marostegui@cumin1004: START - Cookbook sre.mysql.clone of db2168.codfw.wmnet onto db2169.codfw.wmnet
  • 05:21 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2169: moving to s7
  • 05:19 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool db2169: moving to s7
  • 04:01 mwpresync@deploy1003: Pruned MediaWiki: 1.47.0-wmf.20 (duration: 01m 00s)
  • 03:30 mwpresync@deploy1003: Finished scap sync-world: testwikis to 1.47.0-wmf.23 refs T438219 (duration: 27m 17s)
  • 03:28 mwpresync@deploy1003: Rolling back deployment
  • 03:03 mwpresync@deploy1003: Started scap sync-world: testwikis to 1.47.0-wmf.23 refs T438219
  • 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 37s)
  • 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image

2026-10-05

  • 23:24 derenrich@deploy1003: Finished scap sync-world: Backport for Enable the discord previews from WikimediaCustomizations on enwiki (T439914) (duration: 11m 59s)
  • 23:18 derenrich@deploy1003: derenrich: Continuing with deployment
  • 23:16 derenrich@deploy1003: derenrich: Backport for Enable the discord previews from WikimediaCustomizations on enwiki (T439914) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 23:12 derenrich@deploy1003: Started scap sync-world: Backport for Enable the discord previews from WikimediaCustomizations on enwiki (T439914)
  • 22:00 dzahn@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts zuul1003.eqiad.wmnet
  • 22:00 dzahn@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 22:00 dzahn@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: zuul1003.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1004"
  • 22:00 dzahn@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: zuul1003.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1004"
  • 21:53 dzahn@cumin1004: START - Cookbook sre.dns.netbox
  • 21:48 dzahn@cumin1004: START - Cookbook sre.hosts.decommission for hosts zuul1003.eqiad.wmnet
  • 21:45 dzahn@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts zuul1002.eqiad.wmnet
  • 21:45 dzahn@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 21:45 dzahn@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: zuul1002.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1004"
  • 21:45 dzahn@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: zuul1002.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1004"
  • 21:39 jasmine@cumin2003: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=99) for hosts testreduce1002.eqiad.wmnet
  • 21:39 jasmine@cumin2003: START - Cookbook sre.hosts.decommission for hosts testreduce1002.eqiad.wmnet
  • 21:38 dzahn@cumin1004: START - Cookbook sre.dns.netbox
  • 21:35 jasmine@cumin2003: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=99) for hosts testreduce1002.eqiad.wmnet
  • 21:34 jasmine@cumin2003: START - Cookbook sre.hosts.decommission for hosts testreduce1002.eqiad.wmnet
  • 21:33 dzahn@cumin1004: START - Cookbook sre.hosts.decommission for hosts zuul1002.eqiad.wmnet
  • 21:10 bking@cumin2003: END (PASS) - Cookbook sre.hadoop.roll-restart-workers (exit_code=0) restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade.
  • 21:05 bking@cumin2003: START - Cookbook sre.hadoop.roll-restart-workers restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade.
  • 21:02 bking@cumin2003: END (PASS) - Cookbook sre.hadoop.roll-restart-workers (exit_code=0) restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade.
  • 20:58 logmsgbot: dreamyjazz Deployed security patch for T439719
  • 20:50 bking@cumin2003: START - Cookbook sre.hadoop.roll-restart-workers restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade.
  • 20:45 logmsgbot: dreamyjazz Deployed security patch for T439974
  • 20:09 cdobbins@cumin1004: END (PASS) - Cookbook sre.dns.roll-restart-ntp (exit_code=0) rolling restart_daemons on A:dnsbox
  • 20:01 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
  • 20:01 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
  • 19:51 logmsgbot: dreamyjazz Deployed security patch for T439932
  • 19:33 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
  • 19:33 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
  • 19:09 cdobbins@cumin1004: END (PASS) - Cookbook sre.cdn.roll-restart-ats (exit_code=0) rolling restart_daemons on A:cp-esams
  • 18:58 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
  • 18:58 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
  • 18:46 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
  • 18:45 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
  • 18:44 cdobbins@cumin1004: START - Cookbook sre.cdn.roll-restart-ats rolling restart_daemons on A:cp-esams
  • 18:39 cdobbins@cumin1004: END (PASS) - Cookbook sre.cdn.roll-restart-ats (exit_code=0) rolling restart_daemons on A:cp-drmrs
  • 18:23 isaranto@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
  • 18:14 cdobbins@cumin1004: START - Cookbook sre.cdn.roll-restart-ats rolling restart_daemons on A:cp-drmrs
  • 18:08 cdobbins@cumin1004: END (PASS) - Cookbook sre.cdn.roll-restart-ats (exit_code=0) rolling restart_daemons on A:cp-ulsfo
  • 17:58 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply
  • 17:58 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply
  • 17:58 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply
  • 17:57 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply
  • 17:53 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply
  • 17:52 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply
  • 17:46 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply
  • 17:46 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply
  • 17:43 cdobbins@cumin1004: START - Cookbook sre.cdn.roll-restart-ats rolling restart_daemons on A:cp-ulsfo
  • 17:42 sbassett: deployed csv file to security.wikimedia.org (miscweb)
  • 17:41 cdobbins@cumin1004: END (PASS) - Cookbook sre.cdn.roll-restart-ats (exit_code=0) rolling restart_daemons on A:cp-ulsfo
  • 17:40 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply
  • 17:39 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply
  • 17:33 urbanecm@deploy1003: mwscript-k8s job started: GrowthExperiments:reassignMentees --wiki=enwiki --mentor=Maliner --performer=Maliner --as-job # T439577
  • 17:29 swfrench@deploy1003: Finished scap deploy-service shellbox: Test deployment after logging and configuration changes (duration: 01m 31s)
  • 17:29 swfrench@deploy1003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
  • 17:28 swfrench@deploy1003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] codfw Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
  • 17:27 swfrench@deploy1003: scap deploy-service shellbox: [STAGING/DEFAULT] staging-eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
  • 17:27 swfrench@deploy1003: Started scap deploy-service shellbox: Test deployment after logging and configuration changes
  • 17:16 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-jobrunner: apply
  • 17:16 cdobbins@cumin1004: START - Cookbook sre.cdn.roll-restart-ats rolling restart_daemons on A:cp-ulsfo
  • 17:15 cdobbins@cumin1004: END (PASS) - Cookbook sre.cdn.roll-restart-ats (exit_code=0) rolling restart_daemons on A:cp-magru
  • 17:15 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-jobrunner: apply
  • 17:15 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-pretrain: apply
  • 17:15 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-pretrain: apply
  • 17:14 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-jobrunner: apply
  • 17:13 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/mw-jobrunner: apply
  • 17:12 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-pretrain: apply
  • 17:11 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/mw-pretrain: apply
  • 16:50 cdobbins@cumin1004: START - Cookbook sre.cdn.roll-restart-ats rolling restart_daemons on A:cp-magru
  • 16:32 dcausse: closing the "Distributed search queries experiment" window
  • 16:30 dcausse@deploy1003: Finished scap sync-world: Backport for Revert "search: test perf of moving completion out of eqiad" (T438984) (duration: 10m 48s)
  • 16:25 dcausse@deploy1003: dcausse: Continuing with deployment
  • 16:23 dcausse@deploy1003: dcausse: Backport for Revert "search: test perf of moving completion out of eqiad" (T438984) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 16:19 cdobbins@cumin1004: START - Cookbook sre.dns.roll-restart-ntp rolling restart_daemons on A:dnsbox
  • 16:19 dcausse@deploy1003: Started scap sync-world: Backport for Revert "search: test perf of moving completion out of eqiad" (T438984)
  • 16:16 cdobbins@cumin1004: END (PASS) - Cookbook sre.cdn.roll-restart-ats (exit_code=0) rolling restart_daemons on A:cp-eqiad
  • 16:05 dcausse@deploy1003: Finished scap sync-world: Backport for Revert "search: measure impact of enwiki" (T438984), search: test perf of moving completion out of eqiad (T438984) (duration: 10m 44s)
  • 15:59 dcausse@deploy1003: dcausse: Continuing with deployment
  • 15:58 dcausse@deploy1003: dcausse: Backport for Revert "search: measure impact of enwiki" (T438984), search: test perf of moving completion out of eqiad (T438984) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 15:54 dcausse@deploy1003: Started scap sync-world: Backport for Revert "search: measure impact of enwiki" (T438984), search: test perf of moving completion out of eqiad (T438984)
  • 15:51 cdobbins@cumin1004: START - Cookbook sre.cdn.roll-restart-ats rolling restart_daemons on A:cp-eqiad
  • 15:47 dcausse@deploy1003: Finished scap sync-world: Backport for Revert "search: test impact of cross project searches" (T438984), search: measure impact of enwiki (T438984) (duration: 25m 12s)
  • 15:47 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-druid1007.eqiad.wmnet with OS bookworm
  • 15:47 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2039.codfw.wmnet
  • 15:47 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2039.codfw.wmnet
  • 15:35 cdobbins@cumin1004: END (PASS) - Cookbook sre.dns.roll-restart (exit_code=0) rolling restart_daemons on A:dnsbox
  • 15:33 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2039.codfw.wmnet
  • 15:32 elukey@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'sync'.
  • 15:32 elukey@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'sync'.
  • 15:32 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2039.codfw.wmnet
  • 15:32 elukey@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'sync'.
  • 15:32 elukey@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'sync'.
  • 15:32 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2039.codfw.wmnet
  • 15:32 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-druid1007.eqiad.wmnet with reason: host reimage
  • 15:30 elukey@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'sync'.
  • 15:30 elukey@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'sync'.
  • 15:29 elukey@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'sync'.
  • 15:29 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2039.codfw.wmnet
  • 15:29 elukey@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'sync'.
  • 15:29 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on an-druid1007.eqiad.wmnet with reason: host reimage
  • 15:28 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2039.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 15:27 dcausse@deploy1003: dcausse: Continuing with deployment
  • 15:26 dcausse@deploy1003: dcausse: Backport for Revert "search: test impact of cross project searches" (T438984), search: measure impact of enwiki (T438984) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 15:23 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2039.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 15:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2039.codfw.wmnet
  • 15:22 dcausse@deploy1003: Started scap sync-world: Backport for Revert "search: test impact of cross project searches" (T438984), search: measure impact of enwiki (T438984)
  • 15:22 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2039.codfw.wmnet
  • 15:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2038.codfw.wmnet
  • 15:22 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2038.codfw.wmnet
  • 15:21 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2038.codfw.wmnet
  • 15:20 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2038.codfw.wmnet
  • 15:20 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2038.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 15:18 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
  • 15:17 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
  • 15:16 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host an-druid1007.eqiad.wmnet with OS bookworm
  • 15:16 elukey@dns1004: END - running authdns-update
  • 15:15 dcausse@deploy1003: Finished scap sync-world: Backport for Revert "search: test impact of the function score window size" (T438984), search: test impact of cross project searches (T438984) (duration: 10m 11s)
  • 15:14 elukey@dns1004: START - running authdns-update
  • 15:12 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2038.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 15:12 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply
  • 15:12 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply
  • 15:12 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2038.codfw.wmnet
  • 15:12 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply
  • 15:12 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply
  • 15:12 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
  • 15:11 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2038.codfw.wmnet
  • 15:11 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2037.codfw.wmnet
  • 15:10 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2037.codfw.wmnet
  • 15:10 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2037.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 15:10 dcausse@deploy1003: dcausse: Continuing with deployment
  • 15:09 dcausse@deploy1003: dcausse: Backport for Revert "search: test impact of the function score window size" (T438984), search: test impact of cross project searches (T438984) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 15:05 dcausse@deploy1003: Started scap sync-world: Backport for Revert "search: test impact of the function score window size" (T438984), search: test impact of cross project searches (T438984)
  • 15:02 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply
  • 15:02 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2037.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 15:02 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply
  • 15:02 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2037.codfw.wmnet
  • 15:01 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2037.codfw.wmnet
  • 15:01 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2036.codfw.wmnet
  • 15:01 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2036.codfw.wmnet
  • 15:00 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2036.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 15:00 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-druid1006.eqiad.wmnet with OS bookworm
  • 14:58 dcausse@deploy1003: Finished scap sync-world: Backport for Revert "search: test impact of MLR" (T438984), search: test impact of the function score window size (T438984) (duration: 10m 12s)
  • 14:53 dcausse@deploy1003: dcausse: Continuing with deployment
  • 14:53 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply
  • 14:53 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply
  • 14:52 dcausse@deploy1003: dcausse: Backport for Revert "search: test impact of MLR" (T438984), search: test impact of the function score window size (T438984) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 14:52 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2036.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 14:49 osleger@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
  • 14:49 osleger@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
  • 14:49 osleger@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
  • 14:49 osleger@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
  • 14:48 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: apply
  • 14:48 dcausse@deploy1003: Started scap sync-world: Backport for Revert "search: test impact of MLR" (T438984), search: test impact of the function score window size (T438984)
  • 14:48 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: apply
  • 14:48 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2036.codfw.wmnet
  • 14:48 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply
  • 14:47 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply
  • 14:47 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2036.codfw.wmnet
  • 14:47 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2035.codfw.wmnet
  • 14:47 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2035.codfw.wmnet
  • 14:47 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply
  • 14:46 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'.
  • 14:46 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2035.codfw.wmnet
  • 14:46 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply
  • 14:46 dancy@deploy1003: Installation of scap version "4.294.0" completed for 3 hosts
  • 14:46 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'.
  • 14:46 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2035.codfw.wmnet
  • 14:45 osleger@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
  • 14:45 osleger@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
  • 14:45 osleger@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
  • 14:45 osleger@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
  • 14:45 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2035.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 14:44 dancy@deploy1003: Installing scap version "4.294.0" for 3 host(s)
  • 14:44 osleger@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
  • 14:43 osleger@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
  • 14:43 osleger@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
  • 14:43 osleger@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
  • 14:42 dkertesz@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-eqiad and A:cp - upgrade haproxy to 3.2.25-1 (T439667)
  • 14:38 sbassett@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
  • 14:38 dcausse@deploy1003: Finished scap sync-world: Backport for Revert "search: test impact of the MLR window size" (T438984), search: test impact of MLR (T438984) (duration: 10m 04s)
  • 14:38 sbassett@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
  • 14:38 sbassett@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
  • 14:37 sbassett@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
  • 14:37 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-druid1006.eqiad.wmnet with reason: host reimage
  • 14:36 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2035.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 14:36 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2035.codfw.wmnet
  • 14:36 cdobbins@cumin1004: START - Cookbook sre.dns.roll-restart rolling restart_daemons on A:dnsbox
  • 14:35 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2035.codfw.wmnet
  • 14:35 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2034.codfw.wmnet
  • 14:35 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2034.codfw.wmnet
  • 14:34 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2034.codfw.wmnet
  • 14:33 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2034.codfw.wmnet
  • 14:33 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2034.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 14:33 dcausse@deploy1003: dcausse: Continuing with deployment
  • 14:32 dcausse@deploy1003: dcausse: Backport for Revert "search: test impact of the MLR window size" (T438984), search: test impact of MLR (T438984) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 14:32 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on an-druid1006.eqiad.wmnet with reason: host reimage
  • 14:30 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'.
  • 14:30 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'.
  • 14:28 dcausse@deploy1003: Started scap sync-world: Backport for Revert "search: test impact of the MLR window size" (T438984), search: test impact of MLR (T438984)
  • 14:27 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2034.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 14:26 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2034.codfw.wmnet
  • 14:25 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2034.codfw.wmnet
  • 14:25 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2033.codfw.wmnet
  • 14:25 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2033.codfw.wmnet
  • 14:24 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2033.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 14:20 dcausse@deploy1003: Finished scap sync-world: Backport for search: test impact of the MLR window size (T438984) (duration: 11m 12s)
  • 14:20 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host an-druid1006.eqiad.wmnet with OS bookworm
  • 14:20 btullis@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host an-druid1006.eqiad.wmnet with OS bookworm
  • 14:18 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2033.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 14:18 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2033.codfw.wmnet
  • 14:17 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2033.codfw.wmnet
  • 14:16 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2018.codfw.wmnet
  • 14:16 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2018.codfw.wmnet
  • 14:16 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2018.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 14:15 dcausse@deploy1003: dcausse: Continuing with deployment
  • 14:13 dcausse@deploy1003: dcausse: Backport for search: test impact of the MLR window size (T438984) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 14:12 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
  • 14:12 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
  • 14:10 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2018.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 14:09 dcausse@deploy1003: Started scap sync-world: Backport for search: test impact of the MLR window size (T438984)
  • 14:07 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host an-druid1006.eqiad.wmnet with OS bookworm
  • 14:07 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-druid1006.eqiad.wmnet with OS bookworm
  • 14:04 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2018.codfw.wmnet
  • 14:03 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2018.codfw.wmnet
  • 14:03 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync
  • 14:03 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2017.codfw.wmnet
  • 14:03 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2017.codfw.wmnet
  • 14:03 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync
  • 14:03 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2017.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 13:58 logmsgbot: dreamyjazz Deployed security patch for T439968
  • 13:57 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'.
  • 13:57 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2017.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 13:56 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'.
  • 13:54 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'.
  • 13:54 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2017.codfw.wmnet
  • 13:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2017.codfw.wmnet
  • 13:51 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2016.codfw.wmnet
  • 13:51 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2016.codfw.wmnet
  • 13:50 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2016.codfw.wmnet
  • 13:49 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2016.codfw.wmnet
  • 13:48 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2016.codfw.wmnet
  • 13:46 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2016.codfw.wmnet
  • 13:45 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2016.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 13:42 zabe@deploy1003: Finished scap sync-world: Backport for Move reads/writes of wbc_entity_usage to x1 for (fr|ja)wiki (T439753) (duration: 09m 59s)
  • 13:42 elukey: restart pybal on lvs10[20,19] to pick up a new LVS service
  • 13:42 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'.
  • 13:40 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2016.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 13:39 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2016.codfw.wmnet
  • 13:39 tappof@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts kafka-logging2003.codfw.wmnet
  • 13:39 tappof@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 13:39 tappof@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004"
  • 13:39 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2016.codfw.wmnet
  • 13:39 tappof@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004"
  • 13:38 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2015.codfw.wmnet
  • 13:38 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2015.codfw.wmnet
  • 13:38 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2015.codfw.wmnet
  • 13:37 zabe@deploy1003: zabe: Continuing with deployment
  • 13:37 elukey: restart pybal on lvs201[34] to pick up a new LVS service
  • 13:37 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2015.codfw.wmnet
  • 13:36 zabe@deploy1003: zabe: Backport for Move reads/writes of wbc_entity_usage to x1 for (fr|ja)wiki (T439753) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 13:36 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2015.codfw.wmnet
  • 13:35 tappof@cumin1004: START - Cookbook sre.dns.netbox
  • 13:34 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2015.codfw.wmnet
  • 13:34 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2015.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 13:32 zabe@deploy1003: Started scap sync-world: Backport for Move reads/writes of wbc_entity_usage to x1 for (fr|ja)wiki (T439753)
  • 13:32 zabe@deploy1003: Finished scap sync-world: Backport for Enable ReadingList CTA on Minerva for all reading list wikis (T438779), Update pulsating blue dot behavior for anonymous bookmark (T438779) (duration: 11m 23s)
  • 13:28 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2015.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 13:27 zabe@deploy1003: jdlrobson, zabe, annet: Continuing with deployment
  • 13:27 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2015.codfw.wmnet
  • 13:26 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2015.codfw.wmnet
  • 13:26 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2011.codfw.wmnet
  • 13:25 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
  • 13:25 zabe@deploy1003: jdlrobson, zabe, annet: Backport for Enable ReadingList CTA on Minerva for all reading list wikis (T438779), Update pulsating blue dot behavior for anonymous bookmark (T438779) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 13:24 dkertesz@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-eqiad and A:cp - upgrade haproxy to 3.2.25-1 (T439667)
  • 13:24 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
  • 13:22 dkertesz@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-drmrs and A:cp - upgrade haproxy to 3.2.25-1 (T439667)
  • 13:21 trueg@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply
  • 13:21 zabe@deploy1003: Started scap sync-world: Backport for Enable ReadingList CTA on Minerva for all reading list wikis (T438779), Update pulsating blue dot behavior for anonymous bookmark (T438779)
  • 13:20 zabe@deploy1003: Finished scap sync-world: Backport for throttle.php: Lift IP cap for edit-a-thon (T439991) (duration: 14m 48s)
  • 13:20 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2014.codfw.wmnet
  • 13:20 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2014.codfw.wmnet
  • 13:20 oblivian@cumin1004: END (ERROR) - Cookbook sre.hosts.provision (exit_code=97) for host wikikube-worker2014.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 13:20 trueg@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply
  • 13:19 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2011.codfw.wmnet
  • 13:19 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2010.codfw.wmnet
  • 13:19 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
  • 13:14 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
  • 13:12 zabe@deploy1003: zabe, tryvix1509: Continuing with deployment
  • 13:12 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2010.codfw.wmnet
  • 13:12 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2009.codfw.wmnet
  • 13:11 zabe@deploy1003: zabe, tryvix1509: Backport for throttle.php: Lift IP cap for edit-a-thon (T439991) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 13:10 tappof@cumin1004: START - Cookbook sre.hosts.decommission for hosts kafka-logging2003.codfw.wmnet
  • 13:05 tappof@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts kafka-logging2002.codfw.wmnet
  • 13:05 zabe@deploy1003: Started scap sync-world: Backport for throttle.php: Lift IP cap for edit-a-thon (T439991)
  • 13:05 tappof@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 13:05 tappof@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004"
  • 13:05 tappof@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004"
  • 13:04 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2009.codfw.wmnet
  • 13:01 tappof@cumin1004: START - Cookbook sre.dns.netbox
  • 13:01 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'.
  • 12:59 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'.
  • 12:58 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
  • 12:57 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host an-druid1006.eqiad.wmnet with OS bookworm
  • 12:56 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
  • 12:56 tappof@cumin1004: START - Cookbook sre.hosts.decommission for hosts kafka-logging2002.codfw.wmnet
  • 12:55 tappof@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts kafka-logging2001.codfw.wmnet
  • 12:55 tappof@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 12:55 tappof@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004"
  • 12:55 tappof@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004"
  • 12:51 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2014.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 12:50 tappof@cumin1004: START - Cookbook sre.dns.netbox
  • 12:49 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2014.codfw.wmnet
  • 12:49 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=1) for host an-druid1005.eqiad.wmnet with OS bookworm
  • 12:49 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2014.codfw.wmnet
  • 12:48 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2013.codfw.wmnet
  • 12:48 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2013.codfw.wmnet
  • 12:48 trueg@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply
  • 12:47 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2013.codfw.wmnet
  • 12:46 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2013.codfw.wmnet
  • 12:46 trueg@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply
  • 12:46 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2013.codfw.wmnet
  • 12:46 tappof@cumin1004: START - Cookbook sre.hosts.decommission for hosts kafka-logging2001.codfw.wmnet
  • 12:45 trueg@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply
  • 12:45 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2013.codfw.wmnet
  • 12:45 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2013.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 12:39 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2013.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 12:38 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest2001.codfw.wmnet with OS trixie
  • 12:38 elukey@cumin1004: END (FAIL) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=99) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - elukey@cumin1004"
  • 12:38 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for instrument(AccountSetup): expose motivation data for control (T439887) (duration: 55m 26s)
  • 12:35 trueg@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply
  • 12:31 moritzm: installing sqlite3 security updates
  • 12:26 lucaswerkmeister-wmde@deploy1003: volker-e, lucaswerkmeister-wmde: Continuing with deployment
  • 12:17 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-druid1005.eqiad.wmnet with reason: host reimage
  • 12:16 moritzm: installing mesa security updates
  • 12:12 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on an-druid1005.eqiad.wmnet with reason: host reimage
  • 12:01 lucaswerkmeister-wmde@deploy1003: volker-e, lucaswerkmeister-wmde: Backport for instrument(AccountSetup): expose motivation data for control (T439887) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 12:00 jmm@puppetserver1001: conftool action : set/pooled=yes; selector: name=ldap-replica1005.wikimedia.org
  • 12:00 jmm@puppetserver1001: conftool action : set/pooled=yes; selector: name=ldap-replica1006.wikimedia.org
  • 12:00 jmm@puppetserver1001: conftool action : set/pooled=yes; selector: name=ldap-replica2007.wikimedia.org
  • 11:58 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
  • 11:55 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host an-druid1005
  • 11:55 btullis@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host an-druid1005
  • 11:55 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2013.codfw.wmnet
  • 11:54 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2013.codfw.wmnet
  • 11:54 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2012.codfw.wmnet
  • 11:54 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2012.codfw.wmnet
  • 11:54 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2012.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 11:53 jmm@puppetserver1001: conftool action : set/pooled=yes; selector: name=ldap-replica2008.wikimedia.org
  • 11:53 jmm@puppetserver1001: conftool action : set/weight=10; selector: name=ldap-replica1006.wikimedia.org
  • 11:53 jmm@puppetserver1001: conftool action : set/weight=10; selector: name=ldap-replica1005.wikimedia.org
  • 11:53 btullis@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host an-druid1005
  • 11:53 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) an-druid1005.eqiad.wmnet 13.53.64.10.in-addr.arpa 3.1.0.0.3.5.0.0.4.6.0.0.0.1.0.0.8.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
  • 11:53 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache an-druid1005.eqiad.wmnet 13.53.64.10.in-addr.arpa 3.1.0.0.3.5.0.0.4.6.0.0.0.1.0.0.8.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
  • 11:53 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 11:53 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host an-druid1005 - btullis@cumin1004"
  • 11:53 jmm@puppetserver1001: conftool action : set/weight=10; selector: name=ldap-replica2007.wikimedia.org
  • 11:52 jmm@puppetserver1001: conftool action : set/weight=10; selector: name=ldap-replica2008.wikimedia.org
  • 11:52 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host an-druid1005 - btullis@cumin1004"
  • 11:48 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2012.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 11:45 btullis@cumin1004: START - Cookbook sre.dns.netbox
  • 11:45 btullis@cumin1004: START - Cookbook sre.hosts.move-vlan for host an-druid1005
  • 11:44 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host an-druid1005.eqiad.wmnet with OS bookworm
  • 11:44 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2012.codfw.wmnet
  • 11:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2012.codfw.wmnet
  • 11:43 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2011.codfw.wmnet
  • 11:43 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2011.codfw.wmnet
  • 11:43 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2011.codfw.wmnet
  • 11:43 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for instrument(AccountSetup): expose motivation data for control (T439887)
  • 11:42 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2011.codfw.wmnet
  • 11:42 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2011.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 11:41 dkertesz@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-drmrs and A:cp - upgrade haproxy to 3.2.25-1 (T439667)
  • 11:35 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2011.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 11:35 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2011.codfw.wmnet
  • 11:34 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2011.codfw.wmnet
  • 11:34 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2006.codfw.wmnet
  • 11:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2006.codfw.wmnet
  • 11:31 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2006.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 11:29 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
  • 11:28 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
  • 11:27 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2006.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 11:27 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2006.codfw.wmnet
  • 11:26 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2006.codfw.wmnet
  • 11:25 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2005.codfw.wmnet
  • 11:25 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2005.codfw.wmnet
  • 11:25 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-druid1004.eqiad.wmnet with OS bookworm
  • 11:25 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2005.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 11:18 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2005.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 11:14 oblivian@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-worker2005.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 11:14 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2005.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 11:14 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2005.codfw.wmnet
  • 11:13 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2005.codfw.wmnet
  • 11:10 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2002.codfw.wmnet
  • 11:10 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2002.codfw.wmnet
  • 11:09 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2002.codfw.wmnet
  • 11:08 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2002.codfw.wmnet
  • 11:08 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2002.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 11:00 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2002.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 10:58 elukey@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - elukey@cumin1004"
  • 10:57 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'.
  • 10:55 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'.
  • 10:52 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-druid1004.eqiad.wmnet with reason: host reimage
  • 10:52 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2002.codfw.wmnet
  • 10:52 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2002.codfw.wmnet
  • 10:52 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2001.codfw.wmnet
  • 10:51 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2001.codfw.wmnet
  • 10:51 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 10:50 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on an-druid1004.eqiad.wmnet with reason: host reimage
  • 10:46 tappof@deploy1003: Finished scap sync-world: T432444 - Remove kafka-logging200[1-3] (duration: 04m 44s)
  • 10:43 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
  • 10:43 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 10:43 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2001.codfw.wmnet
  • 10:43 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2001.codfw.wmnet
  • 10:42 tappof@deploy1003: Started scap sync-world: T432444 - Remove kafka-logging200[1-3]
  • 10:41 tappof@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'sync'.
  • 10:40 tappof@deploy1003: helmfile [codfw] START helmfile.d/admin 'sync'.
  • 10:40 elukey@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
  • 10:40 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2001.codfw.wmnet
  • 10:40 tappof@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'sync'.
  • 10:40 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2001.codfw.wmnet
  • 10:39 tappof@deploy1003: helmfile [eqiad] START helmfile.d/admin 'sync'.
  • 10:38 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2001.codfw.wmnet (remove config only)
  • 10:38 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest2001.codfw.wmnet (remove config only)
  • 10:37 dkertesz@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-codfw and A:cp - upgrade haproxy to 3.2.25-1 (T439667)
  • 10:34 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db1218.eqiad.wmnet with reason: stress testing
  • 10:33 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1218: stress tests
  • 10:33 marostegui: Stop MariaDB on db1218, db2207 and db2209 for stress testing
  • 10:33 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool db1218: stress tests
  • 10:33 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host an-druid1004.eqiad.wmnet with OS bookworm
  • 10:32 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2209: stress tests
  • 10:32 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool db2209: stress tests
  • 10:32 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2209.codfw.wmnet with reason: stress testing
  • 10:31 marostegui: Stop MariaDB on db2207 and db2209 for stress testing
  • 10:31 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1007.eqiad.wmnet
  • 10:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1007.eqiad.wmnet
  • 10:30 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1007.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART (NO reboot will be performed automatically)
  • 10:29 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2001.codfw.wmnet (push config only)
  • 10:29 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest2001.codfw.wmnet (push config only)
  • 10:29 elukey@cumin1004: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS trixie
  • 10:29 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2207.codfw.wmnet with reason: stress testing
  • 10:28 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-druid1003.eqiad.wmnet with OS bookworm
  • 10:28 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2207: stress tests
  • 10:27 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool db2207: stress tests
  • 10:25 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1007.eqiad.wmnet
  • 10:25 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1007.eqiad.wmnet
  • 10:20 elukey@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS trixie
  • 10:19 elukey@cumin1004: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS trixie
  • 10:16 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
  • 10:16 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2001.codfw.wmnet (remove config only)
  • 10:15 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest2001.codfw.wmnet (remove config only)
  • 10:14 blake@deploy1003: Finished scap sync-world: helm-only deploy for T417800 (duration: 09m 40s)
  • 10:08 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-druid1003.eqiad.wmnet with reason: host reimage
  • 10:06 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on an-druid1003.eqiad.wmnet with reason: host reimage
  • 10:05 blake@deploy1003: Started scap sync-world: helm-only deploy for T417800
  • 10:05 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2001.codfw.wmnet (push config only)
  • 10:05 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest2001.codfw.wmnet (push config only)
  • 10:05 elukey@cumin1004: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
  • 10:01 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1007.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART (NO reboot will be performed automatically)
  • 09:50 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host an-druid1003.eqiad.wmnet with OS bookworm
  • 09:50 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-druid1003.eqiad.wmnet with OS bookworm
  • 09:36 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host zuul1004.eqiad.wmnet
  • 09:30 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host zuul1004.eqiad.wmnet
  • 09:26 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sretest2004.codfw.wmnet
  • 09:20 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host sretest2004.codfw.wmnet
  • 09:17 dkertesz@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-codfw and A:cp - upgrade haproxy to 3.2.25-1 (T439667)
  • 08:58 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host testvm2001.codfw.wmnet with OS bookworm
  • 08:53 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-eqiad@eqiad
  • 08:53 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs
  • 08:52 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs
  • 08:46 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad
  • 08:42 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on testvm2001.codfw.wmnet with reason: host reimage
  • 08:37 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on testvm2001.codfw.wmnet with reason: host reimage
  • 08:35 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-codfw@codfw
  • 08:35 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs
  • 08:34 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs
  • 08:30 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-codfw@codfw
  • 08:22 volans: upgrading cloudcumin2001 in-place to trixie T436667
  • 08:20 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host testvm2001.codfw.wmnet with OS bookworm
  • 07:49 dpogorzelski@dns1004: END - running authdns-update
  • 07:46 dpogorzelski@dns1004: START - running authdns-update
  • 07:42 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync
  • 07:41 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync
  • 07:40 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'sync'.
  • 07:39 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'sync'.
  • 07:18 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'.
  • 07:17 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'.
  • 07:16 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
  • 07:15 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
  • 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 46s)
  • 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image

2026-10-04

  • 16:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
  • 16:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
  • 14:17 osleger@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
  • 14:17 osleger@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
  • 14:17 osleger@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
  • 14:17 osleger@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
  • 13:27 osleger@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
  • 13:27 osleger@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
  • 13:27 osleger@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
  • 13:27 osleger@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
  • 13:25 osleger@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
  • 13:25 osleger@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
  • 13:25 osleger@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
  • 13:25 osleger@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
  • 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 27s)
  • 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image

2026-10-03

  • 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 28s)
  • 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image

2026-10-02

  • 21:10 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host an-druid1003.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 21:01 jclark@cumin1004: START - Cookbook sre.hosts.provision for host an-druid1003.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 20:54 inflatador: bking@an-druid100[4-7] `tune2fs -m0 /dev/mapper/vg0-srv` T440033
  • 20:53 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker2010.codfw.wmnet with OS bookworm
  • 20:49 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker2011.codfw.wmnet with OS bookworm
  • 20:40 jclark@cumin1004: START - Cookbook sre.hosts.reimage for host an-druid1003.eqiad.wmnet with OS bookworm
  • 20:37 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker2010.codfw.wmnet with reason: host reimage
  • 20:31 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker2011.codfw.wmnet with reason: host reimage
  • 20:26 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker2010.codfw.wmnet with reason: host reimage
  • 20:26 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker2011.codfw.wmnet with reason: host reimage
  • 20:23 osleger@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
  • 20:23 osleger@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
  • 20:23 osleger@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
  • 20:23 osleger@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
  • 20:22 osleger@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
  • 20:21 osleger@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
  • 20:21 osleger@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
  • 20:20 osleger@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
  • 20:17 dzahn@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host zuul1004.eqiad.wmnet with OS trixie
  • 20:10 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker2010.codfw.wmnet with OS bookworm
  • 20:10 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker2011.codfw.wmnet with OS bookworm
  • 19:58 dzahn@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on zuul1004.eqiad.wmnet with reason: host reimage
  • 19:54 dzahn@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on zuul1004.eqiad.wmnet with reason: host reimage
  • 19:47 bking@cumin2003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['an-druid1003']
  • 19:46 bking@cumin2003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['an-druid1003']
  • 19:39 dzahn@cumin1004: START - Cookbook sre.hosts.reimage for host zuul1004.eqiad.wmnet with OS trixie
  • 19:37 bking@cumin2003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['an-druid1003']
  • 19:37 bking@cumin2003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['an-druid1003']
  • 19:37 bking@cumin2003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['an-druid1003']
  • 19:33 mutante: zuul1004 - reimage stuck in d-i on leftover software RAID/LVM from its previous life as a ganeti host; wiping old md/LVM metadata and partition tables from the installer shell same fix as on zuul1005 (as also mentioned by Valerie/dcops) - T427353
  • 19:08 dzahn@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host zuul1004.eqiad.wmnet with OS trixie
  • 18:40 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host flink-zk2003.codfw.wmnet
  • 18:36 bking@cumin2003: START - Cookbook sre.hosts.reboot-single for host flink-zk2003.codfw.wmnet
  • 18:03 dzahn@cumin1004: START - Cookbook sre.hosts.reimage for host zuul1004.eqiad.wmnet with OS trixie
  • 18:01 dzahn@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts zuul1001.eqiad.wmnet
  • 18:01 dzahn@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 18:01 dzahn@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: zuul1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1004"
  • 18:01 dzahn@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: zuul1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1004"
  • 18:00 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host flink-zk2002.codfw.wmnet
  • 18:00 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host flink-zk1003.eqiad.wmnet
  • 17:56 bking@cumin2003: START - Cookbook sre.hosts.reboot-single for host flink-zk2002.codfw.wmnet
  • 17:56 bking@cumin2003: START - Cookbook sre.hosts.reboot-single for host flink-zk1003.eqiad.wmnet
  • 17:51 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host flink-zk2001.codfw.wmnet
  • 17:50 dzahn@cumin1004: START - Cookbook sre.dns.netbox
  • 17:47 bking@cumin2003: START - Cookbook sre.hosts.reboot-single for host flink-zk2001.codfw.wmnet
  • 17:46 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host flink-zk1002.eqiad.wmnet
  • 17:45 dzahn@cumin1004: START - Cookbook sre.hosts.decommission for hosts zuul1001.eqiad.wmnet
  • 17:44 dzahn@dns1004: END - running authdns-update
  • 17:42 bking@cumin2003: START - Cookbook sre.hosts.reboot-single for host flink-zk1002.eqiad.wmnet
  • 17:41 dzahn@dns1004: START - running authdns-update
  • 17:39 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.reboot (exit_code=0)
  • 17:39 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host flink-zk1001.eqiad.wmnet
  • 17:35 bking@cumin2003: START - Cookbook sre.hosts.reboot-single for host flink-zk1001.eqiad.wmnet
  • 17:25 bking@cumin2003: START - Cookbook sre.wdqs.reboot
  • 17:23 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.reboot (exit_code=0)
  • 16:44 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker2009.codfw.wmnet with OS bookworm
  • 16:33 kevinbazira@deploy1003: helmfile [ml-serve-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' .
  • 16:31 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'tts-section-generator' for release 'main' .
  • 16:28 kevinbazira@deploy1003: helmfile [ml-staging-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' .
  • 16:28 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker2009.codfw.wmnet with reason: host reimage
  • 16:24 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker2009.codfw.wmnet with reason: host reimage
  • 16:22 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ml-cache2003 to dse-k8s-worker2011
  • 16:22 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ml-cache2002 to dse-k8s-worker2010
  • 16:21 btullis@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker2011
  • 16:21 btullis@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker2011
  • 16:21 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker2011 on all recursors
  • 16:21 btullis@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker2010
  • 16:21 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker2011 on all recursors
  • 16:21 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 16:21 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ml-cache2003 to dse-k8s-worker2011 - btullis@cumin1004"
  • 16:13 btullis@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker2010
  • 16:13 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker2010 on all recursors
  • 16:13 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker2010 on all recursors
  • 16:13 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 16:10 btullis@cumin1004: START - Cookbook sre.dns.netbox
  • 16:09 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ml-cache2003 to dse-k8s-worker2011 - btullis@cumin1004"
  • 16:07 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker2009.codfw.wmnet with OS bookworm
  • 16:05 btullis@cumin1004: START - Cookbook sre.dns.netbox
  • 16:05 btullis@cumin1004: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
  • 16:04 btullis@cumin1004: START - Cookbook sre.hosts.rename from ml-cache2003 to dse-k8s-worker2011
  • 16:03 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ml-cache2001 to dse-k8s-worker2009
  • 16:03 btullis@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker2009
  • 16:03 btullis@cumin1004: START - Cookbook sre.dns.netbox
  • 16:03 btullis@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker2009
  • 16:03 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker2009 on all recursors
  • 16:02 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker2009 on all recursors
  • 16:02 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 16:02 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ml-cache2001 to dse-k8s-worker2009 - btullis@cumin1004"
  • 16:02 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
  • 16:02 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
  • 16:02 btullis@cumin1004: START - Cookbook sre.hosts.rename from ml-cache2002 to dse-k8s-worker2010
  • 16:01 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ml-cache2001 to dse-k8s-worker2009 - btullis@cumin1004"
  • 15:57 btullis@cumin1004: START - Cookbook sre.dns.netbox
  • 15:56 btullis@cumin1004: START - Cookbook sre.hosts.rename from ml-cache2001 to dse-k8s-worker2009
  • 15:52 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-test-client1002.eqiad.wmnet
  • 15:51 bking@cumin2003: START - Cookbook sre.wdqs.reboot
  • 15:48 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host an-test-client1002.eqiad.wmnet
  • 15:47 btullis@cumin1004: END (ERROR) - Cookbook sre.druid.reboot-workers (exit_code=97) for Druid analytics cluster: Reboot Druid nodes
  • 15:45 bking@cumin2003: END (FAIL) - Cookbook sre.wdqs.reboot (exit_code=99)
  • 15:44 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host an-druid1003.eqiad.wmnet with OS bookworm
  • 15:21 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-cache2003.codfw.wmnet with OS bookworm
  • 15:21 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004"
  • 15:20 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004"
  • 15:04 bking@cumin2003: START - Cookbook sre.wdqs.reboot
  • 15:03 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-cache2003.codfw.wmnet with reason: host reimage
  • 15:00 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ml-cache2003.codfw.wmnet with reason: host reimage
  • 14:56 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
  • 14:56 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
  • 14:47 btullis@cumin1004: START - Cookbook sre.druid.reboot-workers for Druid analytics cluster: Reboot Druid nodes
  • 14:45 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-conf1006.eqiad.wmnet
  • 14:44 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host ml-cache2003.codfw.wmnet with OS bookworm
  • 14:39 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host an-conf1006.eqiad.wmnet
  • 14:37 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-conf1005.eqiad.wmnet
  • 14:32 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host an-conf1005.eqiad.wmnet
  • 14:32 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ml-cache2003.codfw.wmnet with OS bookworm
  • 14:31 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host ml-cache2003.codfw.wmnet with OS bookworm
  • 14:30 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-conf1004.eqiad.wmnet
  • 14:25 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ml-cache2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
  • 14:25 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host an-conf1004.eqiad.wmnet
  • 14:23 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
  • 14:23 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.reboot (exit_code=0)
  • 14:17 btullis@cumin1004: END (PASS) - Cookbook sre.zookeeper.roll-restart-zookeeper (exit_code=0) for Zookeeper A:zookeeper-analytics cluster: Roll restart of jvm daemons.
  • 14:10 btullis@cumin1004: START - Cookbook sre.zookeeper.roll-restart-zookeeper for Zookeeper A:zookeeper-analytics cluster: Roll restart of jvm daemons.
  • 14:03 bking@cumin2003: START - Cookbook sre.wdqs.reboot
  • 13:58 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.reboot (exit_code=0)
  • 13:49 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-cache2002.codfw.wmnet with OS bookworm
  • 13:49 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004"
  • 13:36 bking@cumin2003: START - Cookbook sre.wdqs.reboot
  • 12:49 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-cache2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
  • 12:36 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004"
  • 12:27 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
  • 12:20 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-cache2002.codfw.wmnet with reason: host reimage
  • 12:20 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
  • 12:19 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2001.codfw.wmnet (remove config only)
  • 12:19 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest2001.codfw.wmnet (remove config only)
  • 12:16 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ml-cache2002.codfw.wmnet with reason: host reimage
  • 12:15 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 12:15 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache2002 - btullis@cumin1004"
  • 12:15 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache2002 - btullis@cumin1004"
  • 12:14 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2001.codfw.wmnet (push config only)
  • 12:14 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest2001.codfw.wmnet (push config only)
  • 12:14 elukey@cumin1004: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
  • 12:11 elukey@cumin1004: END (ERROR) - Cookbook sre.hosts.provision (exit_code=97) for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
  • 12:11 elukey@cumin1004: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
  • 12:01 btullis@cumin1004: START - Cookbook sre.dns.netbox
  • 12:00 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host ml-cache2002.codfw.wmnet with OS bookworm
  • 12:00 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ml-cache2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
  • 11:58 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
  • 11:56 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-cache2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
  • 11:40 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
  • 11:40 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 11:37 btullis@cumin1004: START - Cookbook sre.dns.netbox
  • 11:08 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-cache2001.codfw.wmnet with OS bookworm
  • 11:08 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004"
  • 10:58 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004"
  • 10:55 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 10:54 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache2002 - btullis@cumin1004"
  • 10:54 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache2002 - btullis@cumin1004"
  • 10:47 btullis@cumin1004: START - Cookbook sre.dns.netbox
  • 10:42 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-cache2001.codfw.wmnet with reason: host reimage
  • 10:39 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ml-cache2001.codfw.wmnet with reason: host reimage
  • 10:29 moritzm: installing Linux 6.12.111 on Trixie hosts
  • 10:23 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host ml-cache2001.codfw.wmnet with OS bookworm
  • 10:22 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-cache2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
  • 10:14 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
  • 10:12 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
  • 10:12 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
  • 10:11 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
  • 10:11 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
  • 09:53 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART (NO reboot will be performed automatically)
  • 09:45 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 09:45 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-commissioning ml-cache2001 - btullis@cumin1004"
  • 09:41 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-commissioning ml-cache2001 - btullis@cumin1004"
  • 09:30 btullis@cumin1004: START - Cookbook sre.dns.netbox
  • 09:06 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
  • 08:56 elukey@cumin1004: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
  • 08:49 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'.
  • 08:48 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'.
  • 08:48 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
  • 08:47 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
  • 07:58 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART (NO reboot will be performed automatically)
  • 07:53 oblivian@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-worker1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART (NO reboot will be performed automatically)
  • 07:52 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART (NO reboot will be performed automatically)
  • 07:52 oblivian@cumin1004: END (ERROR) - Cookbook sre.hosts.provision (exit_code=97) for host wikikube-worker1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART (NO reboot will be performed automatically)
  • 07:47 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART (NO reboot will be performed automatically)
  • 06:55 jmm@cumin2003: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Dima koushha out of all services on: 2448 hosts
  • 02:39 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1238.eqiad.wmnet with reason: maint
  • 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 36s)
  • 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image
  • 00:54 dzahn@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
  • 00:54 dzahn@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
  • 00:54 dzahn@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
  • 00:53 dzahn@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
  • 00:53 dzahn@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
  • 00:52 dzahn@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
  • 00:39 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on P{cp11[08,10,12,14].eqiad.wmnet} and A:cp - 7.1.1-2~bpo13+wmf3 ()

2026-10-01

  • 23:38 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp1108.*
  • 23:04 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P{cp11[08,10,12,14].eqiad.wmnet} and A:cp - 7.1.1-2~bpo13+wmf3 ()
  • 21:56 dreamyjazz@deploy1003: Finished scap sync-world: Backport for WikimediaAntiAbuse: Enable personal info content policy on dewiki (T439928) (duration: 10m 19s)
  • 21:51 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment
  • 21:50 dreamyjazz@deploy1003: dreamyjazz: Backport for WikimediaAntiAbuse: Enable personal info content policy on dewiki (T439928) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 21:46 dreamyjazz@deploy1003: Started scap sync-world: Backport for WikimediaAntiAbuse: Enable personal info content policy on dewiki (T439928)
  • 21:33 tappof: Rebalancing has been completed on kafka-logging in codfw: kafka-logging200[1-3] are no longer assigned any partitions — T432444
  • 21:24 egardner@deploy1003: Finished scap sync-world: Backport for Preferred Sources: Move all eligibility checks to the client (T436692) (duration: 11m 02s)
  • 21:19 egardner@deploy1003: egardner: Continuing with deployment
  • 21:17 egardner@deploy1003: egardner: Backport for Preferred Sources: Move all eligibility checks to the client (T436692) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 21:13 egardner@deploy1003: Started scap sync-world: Backport for Preferred Sources: Move all eligibility checks to the client (T436692)
  • 21:06 brennen@deploy1003: Finished scap sync-world: Backport for Fix UTF-8 corruption in newline sanitization (T431960 T426631) (duration: 10m 43s)
  • 21:00 brennen@deploy1003: brennen, srapoj: Continuing with deployment
  • 20:59 brennen@deploy1003: brennen, srapoj: Backport for Fix UTF-8 corruption in newline sanitization (T431960 T426631) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 20:55 brennen@deploy1003: Started scap sync-world: Backport for Fix UTF-8 corruption in newline sanitization (T431960 T426631)
  • 20:55 tappof: Rebalancing is in progress on kafka-logging in codfw to evacuate kafka-logging2001, kafka-logging2002 and kafka-logging2003 - T432444
  • 20:53 bking@cumin2003: END (PASS) - Cookbook sre.hadoop.roll-restart-workers (exit_code=0) restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade.
  • 20:48 mlitn@deploy1003: Finished scap sync-world: Backport for Lose DB call for page prop (T439182), Extract images in parser post-processing (T439182), Make carousel open MMV directly (T439611), Removes image preview from Treatment 1-2 (T439611) (duration: 10m 56s)
  • 20:46 bking@cumin2003: START - Cookbook sre.hadoop.roll-restart-workers restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade.
  • 20:45 bking@cumin2003: END (PASS) - Cookbook sre.hadoop.roll-restart-workers (exit_code=0) restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade.
  • 20:44 bking@cumin2003: START - Cookbook sre.hadoop.roll-restart-workers restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade.
  • 20:43 mlitn@deploy1003: mlitn, ksarabia: Continuing with deployment
  • 20:42 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
  • 20:42 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
  • 20:41 mlitn@deploy1003: mlitn, ksarabia: Backport for Lose DB call for page prop (T439182), Extract images in parser post-processing (T439182), Make carousel open MMV directly (T439611), Removes image preview from Treatment 1-2 (T439611) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 20:37 mlitn@deploy1003: Started scap sync-world: Backport for Lose DB call for page prop (T439182), Extract images in parser post-processing (T439182), Make carousel open MMV directly (T439611), Removes image preview from Treatment 1-2 (T439611)
  • 20:35 bking@cumin2003: END (PASS) - Cookbook sre.hadoop.roll-restart-workers (exit_code=0) restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade.
  • 20:34 bking@cumin2003: START - Cookbook sre.hadoop.roll-restart-workers restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade.
  • 20:33 cjming@deploy1003: Finished scap sync-world: Backport for fix(ReadingRecommendations): Update button style to primary (T436332), instrument(WelcomeSurvey): align motivation event with AccountSetup (T439887), instrument(AccountSetup): Record setup motivation on `page_visit` (T439217) (duration: 18m 28s)
  • 20:31 bking@cumin2003: END (PASS) - Cookbook sre.hadoop.roll-restart-workers (exit_code=0) restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade.
  • 20:27 cjming@deploy1003: cjming, volker-e: Continuing with deployment
  • 20:27 bking@cumin2003: START - Cookbook sre.hadoop.roll-restart-workers restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade.
  • 20:26 bking@cumin2003: END (ERROR) - Cookbook sre.hadoop.roll-restart-workers (exit_code=97) restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade.
  • 20:21 bking@cumin2003: START - Cookbook sre.hadoop.roll-restart-workers restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade.
  • 20:18 cjming@deploy1003: cjming, volker-e: Backport for fix(ReadingRecommendations): Update button style to primary (T436332), instrument(WelcomeSurvey): align motivation event with AccountSetup (T439887), instrument(AccountSetup): Record setup motivation on `page_visit` (T439217) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can
  • 20:14 cjming@deploy1003: Started scap sync-world: Backport for fix(ReadingRecommendations): Update button style to primary (T436332), instrument(WelcomeSurvey): align motivation event with AccountSetup (T439887), instrument(AccountSetup): Record setup motivation on `page_visit` (T439217)
  • 20:02 jhathaway: flipping flag to disable legacy facts in puppet prod, attempt two
  • 19:52 zabe@deploy1003: Finished scap sync-world: Backport for Add missing select of fr_timestamp (duration: 09m 40s)
  • 19:51 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
  • 19:51 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
  • 19:46 zabe@deploy1003: zabe: Continuing with deployment
  • 19:46 zabe@deploy1003: zabe: Backport for Add missing select of fr_timestamp synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 19:42 zabe@deploy1003: Started scap sync-world: Backport for Add missing select of fr_timestamp
  • 19:42 zabe@deploy1003: Finished scap sync-world: Backport for Move reads/writes of wbc_entity_usage to x1 for s5 wikis (T439753) (duration: 10m 01s)
  • 19:36 zabe@deploy1003: zabe: Continuing with deployment
  • 19:36 zabe@deploy1003: zabe: Backport for Move reads/writes of wbc_entity_usage to x1 for s5 wikis (T439753) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 19:32 zabe@deploy1003: Started scap sync-world: Backport for Move reads/writes of wbc_entity_usage to x1 for s5 wikis (T439753)
  • 19:19 jhathaway: puppet re-enabled
  • 19:02 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
  • 19:02 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
  • 18:42 brett@cumin1004: END (FAIL) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=1) rolling upgrade of Varnish on A:cp-text_eqiad - 7.1.1-2~bpo13+wmf3 ()
  • 18:40 brennen@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.47.0-wmf.22 refs T438218
  • 18:36 jhathaway: disabling puppet fleet wide to test https://gerrit.wikimedia.org/r/c/operations/puppet/+/1347689
  • 18:25 brennen: 1.47.0-wmf.22 train status (T438218): logs kind of a mess, but no current blockers, rolling to all wikis
  • 18:16 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
  • 18:16 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
  • 18:12 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-upload_eqiad - 7.1.1-2~bpo13+wmf3 ()
  • 17:55 swfrench@deploy1003: Finished scap deploy-service shellbox: Test deploy-service configuration for Shellbox (duration: 03m 28s)
  • 17:55 spiderpig@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply
  • 17:55 spiderpig@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-syntaxhighlight: apply
  • 17:55 spiderpig@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-media: apply
  • 17:54 spiderpig@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-timeline: apply
  • 17:54 spiderpig@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-constraints: apply
  • 17:54 spiderpig@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox: apply
  • 17:54 spiderpig@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-video: apply
  • 17:54 spiderpig@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-syntaxhighlight: apply
  • 17:54 spiderpig@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox: apply
  • 17:54 spiderpig@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-media: apply
  • 17:54 spiderpig@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-timeline: apply
  • 17:54 spiderpig@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-constraints: apply
  • 17:53 spiderpig@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: apply
  • 17:53 spiderpig@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-media: apply
  • 17:53 spiderpig@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-constraints: apply
  • 17:53 spiderpig@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply
  • 17:53 spiderpig@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-video: apply
  • 17:53 spiderpig@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox: apply
  • 17:53 spiderpig@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox: apply
  • 17:53 spiderpig@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply
  • 17:53 spiderpig@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-video: apply
  • 17:53 spiderpig@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-constraints: apply
  • 17:53 spiderpig@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-media: apply
  • 17:53 spiderpig@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: apply
  • 17:52 spiderpig@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: apply
  • 17:52 spiderpig@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-video: apply
  • 17:52 spiderpig@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-constraints: apply
  • 17:52 spiderpig@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-syntaxhighlight: apply
  • 17:52 spiderpig@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-media: apply
  • 17:52 spiderpig@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: apply
  • 17:52 spiderpig@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-video: apply
  • 17:52 spiderpig@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-constraints: apply
  • 17:52 spiderpig@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-media: apply
  • 17:52 spiderpig@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: apply
  • 17:52 spiderpig@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: apply
  • 17:52 spiderpig@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-syntaxhighlight: apply
  • 17:52 swfrench@deploy1003: Started scap deploy-service shellbox: Test deploy-service configuration for Shellbox
  • 17:49 kamila@deploy1003: Finished scap sync-world: rebuild for T439100 (duration: 33m 25s)
  • 17:16 kamila@deploy1003: Started scap sync-world: rebuild for T439100
  • 17:13 tappof@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'sync'.
  • 17:12 tappof@deploy1003: helmfile [codfw] START helmfile.d/admin 'sync'.
  • 17:12 tappof@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'sync'.
  • 17:12 tappof@deploy1003: helmfile [eqiad] START helmfile.d/admin 'sync'.
  • 17:01 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
  • 17:01 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
  • 16:25 cdobbins@cumin1004: END (PASS) - Cookbook sre.dns.roll-restart-reboot-durum (exit_code=0) rolling restart_daemons on A:durum and A:durum
  • 16:12 cdobbins@cumin1004: START - Cookbook sre.dns.roll-restart-reboot-durum rolling restart_daemons on A:durum and A:durum
  • 15:40 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-upload_eqiad - 7.1.1-2~bpo13+wmf3 ()
  • 15:39 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-text_eqiad - 7.1.1-2~bpo13+wmf3 ()
  • 15:39 cdobbins@cumin1004: END (PASS) - Cookbook sre.cdn.roll-restart-reboot-tcp-proxy (exit_code=0) rolling restart_daemons on A:tcpproxy and A:tcpproxy
  • 15:32 moritzm: installing socat security updates
  • 15:29 dpogorzelski@dns1004: END - running authdns-update
  • 15:26 cdobbins@cumin1004: START - Cookbook sre.cdn.roll-restart-reboot-tcp-proxy rolling restart_daemons on A:tcpproxy and A:tcpproxy
  • 15:25 dpogorzelski@dns1004: START - running authdns-update
  • 15:20 zabe@deploy1003: Finished scap sync-world: Backport for Add missing alias for filerevision, Add missing alias for filerevision (duration: 10m 12s)
  • 15:14 zabe@deploy1003: zabe: Continuing with deployment
  • 15:14 zabe@deploy1003: zabe: Backport for Add missing alias for filerevision, Add missing alias for filerevision synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 15:11 moritzm: installing postgresql security updates
  • 15:09 zabe@deploy1003: Started scap sync-world: Backport for Add missing alias for filerevision, Add missing alias for filerevision
  • 15:08 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=tcp-proxy.*
  • 15:08 cdobbins@cumin1004: conftool action : set/pooled=no; selector: name=tcp-proxy.*
  • 14:56 moritzm: installing initramfs-tools bugfix updates from trixie point release
  • 14:48 moritzm: installing busybox bugfix updates from trixie point release
  • 14:47 urbanecm@deploy1003: Finished scap sync-world: Backport for NotificationController: Use lazyPush() for the delete job (T439626), NotificationController: Use lazyPush() for the delete job (T439626) (duration: 10m 05s)
  • 14:42 urbanecm@deploy1003: urbanecm: Continuing with deployment
  • 14:42 moritzm: installing zsh bugfix update from trixie point release
  • 14:41 urbanecm@deploy1003: urbanecm: Backport for NotificationController: Use lazyPush() for the delete job (T439626), NotificationController: Use lazyPush() for the delete job (T439626) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 14:37 urbanecm@deploy1003: Started scap sync-world: Backport for NotificationController: Use lazyPush() for the delete job (T439626), NotificationController: Use lazyPush() for the delete job (T439626)
  • 14:32 moritzm: installing e2fsprogs bugfix update from trixie point release
  • 14:31 dpogorzelski@dns1004: END - running authdns-update
  • 14:29 dpogorzelski@dns1004: START - running authdns-update
  • 14:19 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-eqiad@eqiad
  • 14:19 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs
  • 14:18 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs
  • 14:14 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad
  • 14:10 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-codfw@codfw
  • 14:10 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs
  • 14:09 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs
  • 14:06 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-codfw@codfw
  • 14:04 awight: EU afternoon backport complete
  • 14:03 awight@deploy1003: Finished scap sync-world: Backport for Revert "Use explicit numbers in reference lists and Parsoid rendering" (T439790) (duration: 10m 26s)
  • 14:01 dkertesz@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-eqsin and A:cp - upgrade haproxy to 3.2.25-1 (T439667)
  • 13:58 awight@deploy1003: lilients, awight: Continuing with deployment
  • 13:57 awight@deploy1003: lilients, awight: Backport for Revert "Use explicit numbers in reference lists and Parsoid rendering" (T439790) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 13:53 awight@deploy1003: Started scap sync-world: Backport for Revert "Use explicit numbers in reference lists and Parsoid rendering" (T439790)
  • 13:48 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply
  • 13:48 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply
  • 13:47 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply
  • 13:47 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply
  • 13:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply
  • 13:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply
  • 13:43 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply
  • 13:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply
  • 13:41 awight@deploy1003: Finished scap sync-world: Backport for instrument(AccountSetup): don't double count finish/skip clicks, stats(AccountSetup): Record onboarding ready time in Prometheus (T432810), instrument(AccountSetup): Send exposure from server, send ready event (T432164) (duration: 22m 53s)
  • 13:41 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply
  • 13:40 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply
  • 13:36 awight@deploy1003: awight, migr: Continuing with deployment
  • 13:34 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply
  • 13:34 filippo@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudcephosd1055.eqiad.wmnet with OS bookworm
  • 13:33 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wmde: apply
  • 13:33 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wmde: apply
  • 13:33 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wmde: apply
  • 13:32 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply
  • 13:32 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wmde: apply
  • 13:32 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wikidata: apply
  • 13:32 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply
  • 13:31 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wikidata: apply
  • 13:31 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply
  • 13:31 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-sre: apply
  • 13:30 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-sre: apply
  • 13:30 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-search: apply
  • 13:30 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply
  • 13:30 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
  • 13:30 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
  • 13:30 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-search: apply
  • 13:29 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
  • 13:29 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
  • 13:26 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-research: apply
  • 13:26 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative: apply
  • 13:26 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative: apply
  • 13:25 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-research: apply
  • 13:25 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative: apply
  • 13:25 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative: apply
  • 13:25 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-platform-eng: apply
  • 13:25 oblivian@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
  • 13:24 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-platform-eng: apply
  • 13:24 oblivian@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
  • 13:24 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
  • 13:24 oblivian@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
  • 13:24 oblivian@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: sync
  • 13:24 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply
  • 13:23 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative: apply
  • 13:23 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative: apply
  • 13:23 awight@deploy1003: awight, migr: Backport for instrument(AccountSetup): don't double count finish/skip clicks, stats(AccountSetup): Record onboarding ready time in Prometheus (T432810), instrument(AccountSetup): Send exposure from server, send ready event (T432164) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be ve
  • 13:23 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-fr-tech: apply
  • 13:22 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-fr-tech: apply
  • 13:22 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-experiment-platform: apply
  • 13:21 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-experiment-platform: apply
  • 13:21 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
  • 13:20 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
  • 13:19 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-product: apply
  • 13:19 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-product: apply
  • 13:18 awight@deploy1003: Started scap sync-world: Backport for instrument(AccountSetup): don't double count finish/skip clicks, stats(AccountSetup): Record onboarding ready time in Prometheus (T432810), instrument(AccountSetup): Send exposure from server, send ready event (T432164)
  • 13:18 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative: apply
  • 13:17 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative: apply
  • 13:12 jmm@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe
  • 13:08 jmm@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe
  • 13:08 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply
  • 13:07 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply
  • 13:07 oblivian@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: apply
  • 13:06 oblivian@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: apply
  • 13:05 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 13:05 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add VIPs for ldap-ro-test in eqiad/codfw - jmm@cumin2003"
  • 12:59 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply
  • 12:59 mszwarc@deploy1003: Synchronized private/PrivateSettings.php: Sync PrivateSettings.php (duration: 10m 02s)
  • 12:59 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply
  • 12:54 filippo@cumin1004: START - Cookbook sre.hosts.reimage for host cloudcephosd1055.eqiad.wmnet with OS bookworm
  • 12:54 mszwarc@deploy1003: mszwarc: Continuing with deployment
  • 12:53 mszwarc@deploy1003: mszwarc: Sync PrivateSettings.php synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 12:45 ihurbain@deploy1003: helmfile [codfw] DONE helmfile.d/services/mobileapps: apply
  • 12:44 ihurbain@deploy1003: helmfile [codfw] START helmfile.d/services/mobileapps: apply
  • 12:44 ihurbain@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply
  • 12:40 ihurbain@deploy1003: helmfile [eqiad] START helmfile.d/services/mobileapps: apply
  • 12:34 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
  • 12:34 moritzm: installing libtasn1-6 security updates
  • 12:33 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
  • 12:32 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
  • 12:27 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-pretrain: apply
  • 12:27 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-pretrain: apply
  • 12:25 a-pizzata@deploy1003: Finished deploy [analytics/refinery@1337993]: Regular analytics weekly train [analytics/refinery@13379939] (duration: 04m 27s)
  • 12:24 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
  • 12:23 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
  • 12:22 moritzm: installing glibc security updates
  • 12:21 a-pizzata@deploy1003: Started deploy [analytics/refinery@1337993]: Regular analytics weekly train [analytics/refinery@13379939]
  • 12:20 a-pizzata@deploy1003: Finished deploy [analytics/refinery@1337993] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@13379939] (duration: 00m 46s)
  • 12:19 a-pizzata@deploy1003: Started deploy [analytics/refinery@1337993] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@13379939]
  • 12:15 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-pretrain: apply
  • 12:14 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-pretrain: apply
  • 12:06 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
  • 12:05 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
  • 12:05 dkertesz@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-eqsin and A:cp - upgrade haproxy to 3.2.25-1 (T439667)
  • 11:58 jmm@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe-eqiad
  • 11:55 jforrester@deploy1003: Finished scap sync-world: Backport for RepoHooks: Fix fresh-result rate limit blocking autoconfirmed users (T437533) (duration: 12m 27s)
  • 11:50 jforrester@deploy1003: jforrester: Continuing with deployment
  • 11:50 jmm@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe-eqiad
  • 11:47 jforrester@deploy1003: jforrester: Backport for RepoHooks: Fix fresh-result rate limit blocking autoconfirmed users (T437533) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 11:43 jforrester@deploy1003: Started scap sync-world: Backport for RepoHooks: Fix fresh-result rate limit blocking autoconfirmed users (T437533)
  • 11:36 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add VIPs for ldap-ro-test in eqiad/codfw - jmm@cumin2003"
  • 11:32 jmm@cumin2003: START - Cookbook sre.dns.netbox
  • 11:31 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync
  • 11:30 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync
  • 11:30 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
  • 11:29 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'sync'.
  • 11:28 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'sync'.
  • 11:06 jmm@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe-codfw
  • 10:57 jmm@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe-codfw
  • 10:50 javiermonton@deploy1003: Finished scap sync-world: Backport for stream: mediawiki.job.pageviewTrendingRelativeUpdate (T437875) (duration: 10m 51s)
  • 10:45 javiermonton@deploy1003: javiermonton: Continuing with deployment
  • 10:44 javiermonton@deploy1003: javiermonton: Backport for stream: mediawiki.job.pageviewTrendingRelativeUpdate (T437875) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 10:40 javiermonton@deploy1003: Started scap sync-world: Backport for stream: mediawiki.job.pageviewTrendingRelativeUpdate (T437875)
  • 10:30 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
  • 10:30 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
  • 10:24 ammarpad@deploy1003: mwscript-k8s job started: extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=wikidatawiki --logwiki=metawiki Jakehekd Ivyeclipse # T439805
  • 10:23 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
  • 10:22 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
  • 10:22 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
  • 09:59 dkertesz@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-ulsfo and A:cp - upgrade haproxy to 3.2.25-1 (T439667)
  • 09:47 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
  • 09:47 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
  • 09:35 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
  • 09:35 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
  • 08:39 hashar@deploy1003: Finished deploy [integration/docroot@5cf09c8]: build: Updating brace-expansion to 1.1.21, 2.1.7 (duration: 00m 10s)
  • 08:38 hashar@deploy1003: Started deploy [integration/docroot@5cf09c8]: build: Updating brace-expansion to 1.1.21, 2.1.7
  • 08:35 dkertesz@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-ulsfo and A:cp - upgrade haproxy to 3.2.25-1 (T439667)
  • 08:22 moritzm: restart FPM on phab1005 to pick up openssl/PCRE security updates
  • 08:05 moritzm: installing pcre2/openssl security updates
  • 07:54 vgutierrez: upload purged 0.25 to trixie-wikimedia (apt.wm.o)
  • 07:31 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
  • 07:29 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
  • 07:28 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
  • 07:24 kharlan@deploy1003: Finished scap sync-world: Backport for CheckUser: Enable ClientHintsInEditRequest on all wikis (T401808), CheckUser: Enable ClientHintsInEditRequest on group1 (T401808) (duration: 14m 37s)
  • 07:23 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply
  • 07:16 kharlan@deploy1003: kharlan: Continuing with deployment
  • 07:15 kharlan@deploy1003: kharlan: Backport for CheckUser: Enable ClientHintsInEditRequest on all wikis (T401808), CheckUser: Enable ClientHintsInEditRequest on group1 (T401808) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 07:11 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
  • 07:09 kharlan@deploy1003: Started scap sync-world: Backport for CheckUser: Enable ClientHintsInEditRequest on all wikis (T401808), CheckUser: Enable ClientHintsInEditRequest on group1 (T401808)
  • 07:07 jmm@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply
  • 07:06 kharlan@deploy1003: Finished scap sync-world: Backport for Add extra UI elements to Special:AbuseReview following user feedback (T439694) (duration: 33m 18s)
  • 06:57 tappof@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts kafka-logging1002.eqiad.wmnet
  • 06:57 tappof@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 06:57 tappof@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging1002.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004"
  • 06:57 tappof@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging1002.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004"
  • 06:54 kharlan@deploy1003: kharlan: Continuing with deployment
  • 06:53 tappof@cumin1004: START - Cookbook sre.dns.netbox
  • 06:53 kharlan@deploy1003: kharlan: Backport for Add extra UI elements to Special:AbuseReview following user feedback (T439694) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 06:47 tappof@cumin1004: START - Cookbook sre.hosts.decommission for hosts kafka-logging1002.eqiad.wmnet
  • 06:46 tappof@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts kafka-logging1001.eqiad.wmnet
  • 06:46 tappof@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 06:46 tappof@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004"
  • 06:45 tappof@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004"
  • 06:41 tappof@cumin1004: START - Cookbook sre.dns.netbox
  • 06:35 tappof@cumin1004: START - Cookbook sre.hosts.decommission for hosts kafka-logging1001.eqiad.wmnet
  • 06:33 kharlan@deploy1003: Started scap sync-world: Backport for Add extra UI elements to Special:AbuseReview following user feedback (T439694)
  • 06:29 kharlan@deploy1003: Finished scap sync-world: Backport for AbuseReview: Send the abuse review tag in interaction events (T439238), AbuseReview: Instrument when content policy match adds tags (T439242) (duration: 10m 33s)
  • 06:23 kharlan@deploy1003: kharlan: Continuing with deployment
  • 06:22 kharlan@deploy1003: kharlan: Backport for AbuseReview: Send the abuse review tag in interaction events (T439238), AbuseReview: Instrument when content policy match adds tags (T439242) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 06:18 kharlan@deploy1003: Started scap sync-world: Backport for AbuseReview: Send the abuse review tag in interaction events (T439238), AbuseReview: Instrument when content policy match adds tags (T439242)

2026-09-30

  • 22:23 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-text_codfw - 7.1.1-2~bpo13+wmf3 ()
  • 21:22 sbassett: Deployed updated mitigation for T435455
  • 21:09 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply
  • 21:09 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply
  • 21:00 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply
  • 21:00 arlolra@deploy1003: Finished scap sync-world: Backport for Set a default for wgSplitParsoidParserCache in prod (T432760) (duration: 12m 28s)
  • 21:00 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply
  • 20:58 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply
  • 20:57 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply
  • 20:52 arlolra@deploy1003: arlolra: Continuing with deployment
  • 20:52 arlolra@deploy1003: arlolra: Backport for Set a default for wgSplitParsoidParserCache in prod (T432760) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 20:47 arlolra@deploy1003: Started scap sync-world: Backport for Set a default for wgSplitParsoidParserCache in prod (T432760)
  • 20:44 mstyles@deploy1003: Finished scap sync-world: Backport for Email confirmation: implement delayed enforcement experiment (T439283), Email confirmation: Add server-side enforcement (T439283), Email confirmation: Prepare for next experiment (T439283) (duration: 32m 47s)
  • 20:38 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply
  • 20:38 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply
  • 20:32 mstyles@deploy1003: mstyles, catrope: Continuing with deployment
  • 20:31 mstyles@deploy1003: mstyles, catrope: Backport for Email confirmation: implement delayed enforcement experiment (T439283), Email confirmation: Add server-side enforcement (T439283), Email confirmation: Prepare for next experiment (T439283) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 20:20 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply
  • 20:20 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply
  • 20:20 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply
  • 20:11 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply
  • 20:11 mstyles@deploy1003: Started scap sync-world: Backport for Email confirmation: implement delayed enforcement experiment (T439283), Email confirmation: Add server-side enforcement (T439283), Email confirmation: Prepare for next experiment (T439283)
  • 20:06 kemayo@deploy1003: helmfile [staging] DONE helmfile.d/services/editcheck-headless: apply
  • 20:05 kemayo@deploy1003: helmfile [staging] START helmfile.d/services/editcheck-headless: apply
  • 19:50 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-upload_codfw - 7.1.1-2~bpo13+wmf3 ()
  • 19:45 jhathaway: reverting legacy facts disablement, discovered some remaining usage
  • 18:56 dzahn@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
  • 18:56 dzahn@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
  • 18:56 dzahn@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
  • 18:55 dzahn@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
  • 18:55 dzahn@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
  • 18:55 dzahn@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
  • 18:52 brennen@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.22 refs T438218
  • 18:40 brennen@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.47.0-wmf.22 refs T438218
  • 18:37 jhathaway: flipping flag to disable legacy facts in puppet prod
  • 18:24 brennen@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.22 refs T438218
  • 18:22 brennen@deploy1003: Rolling back deployment
  • 18:10 brennen: 1.47.0-wmf.22 train status (T438218): no current blockers, rolling to group1
  • 17:47 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.restart (exit_code=0)
  • 17:34 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
  • 17:34 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
  • 17:33 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
  • 17:33 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
  • 17:28 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.restart (exit_code=0)
  • 17:21 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-upload_codfw - 7.1.1-2~bpo13+wmf3 ()
  • 17:21 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
  • 17:21 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
  • 17:21 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-text_codfw - 7.1.1-2~bpo13+wmf3 ()
  • 17:18 ryankemper@cumin2003: START - Cookbook sre.wdqs.restart
  • 17:18 ryankemper@cumin2003: START - Cookbook sre.wdqs.restart
  • 17:15 bking@dns2004: END - running authdns-update
  • 17:13 bking@dns2004: START - running authdns-update
  • 17:09 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
  • 17:08 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
  • 16:52 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
  • 16:51 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
  • 16:38 andrew@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephosd1043.eqiad.wmnet with OS bookworm
  • 16:14 aaron@deploy1003: helmfile [eqiad] DONE helmfile.d/services/developer-portal: apply
  • 16:14 aaron@deploy1003: helmfile [eqiad] START helmfile.d/services/developer-portal: apply
  • 16:13 aaron@deploy1003: helmfile [codfw] DONE helmfile.d/services/developer-portal: apply
  • 16:13 aaron@deploy1003: helmfile [codfw] START helmfile.d/services/developer-portal: apply
  • 16:12 andrew@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephosd1043.eqiad.wmnet with reason: host reimage
  • 16:11 aaron@deploy1003: helmfile [staging] DONE helmfile.d/services/developer-portal: apply
  • 16:11 aaron@deploy1003: helmfile [staging] START helmfile.d/services/developer-portal: apply
  • 16:08 andrew@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephosd1043.eqiad.wmnet with reason: host reimage
  • 15:49 andrew@cumin2003: START - Cookbook sre.hosts.reimage for host cloudcephosd1043.eqiad.wmnet with OS bookworm
  • 15:47 andrew@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudcephosd1043.eqiad.wmnet with OS bookworm
  • 15:47 andrew@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephosd1043.eqiad.wmnet with reason: host reimage
  • 15:42 andrew@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephosd1043.eqiad.wmnet with reason: host reimage
  • 15:41 oblivian@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-timeline: apply
  • 15:41 oblivian@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-timeline: apply
  • 15:25 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
  • 15:25 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
  • 15:15 andrew@cumin2003: START - Cookbook sre.hosts.reimage for host cloudcephosd1043.eqiad.wmnet with OS bookworm
  • 15:11 jmm@cumin2003: END (PASS) - Cookbook sre.cdn.roll-restart-reboot-ncredir (exit_code=0) rolling restart_daemons on A:ncredir
  • 15:09 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on D{wikikube-worker236[5-9].codfw.wmnet} and (A:wikikube-master-codfw or A:wikikube-worker-codfw)
  • 15:09 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2369.codfw.wmnet
  • 15:09 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2369.codfw.wmnet
  • 15:09 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
  • 15:09 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
  • 15:03 jmm@cumin2003: START - Cookbook sre.cdn.roll-restart-reboot-ncredir rolling restart_daemons on A:ncredir
  • 15:02 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2369.codfw.wmnet
  • 15:02 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on D{wikikube-worker133[0-4].eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad)
  • 15:02 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker1334.eqiad.wmnet
  • 15:02 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1334.eqiad.wmnet
  • 15:01 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2369.codfw.wmnet
  • 15:01 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2368.codfw.wmnet
  • 15:01 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2368.codfw.wmnet
  • 15:01 jmm@cumin2003: END (PASS) - Cookbook sre.misc-clusters.roll-restart-reboot-docker-registry (exit_code=0) rolling restart_daemons on A:docker-registry
  • 14:58 jmm@cumin2003: START - Cookbook sre.misc-clusters.roll-restart-reboot-docker-registry rolling restart_daemons on A:docker-registry
  • 14:54 zabe@deploy1003: Finished scap sync-world: Backport for Add script to drop duplicate oldimage/filerevision rows (T67264 T428406), Add script to drop duplicate oldimage/filerevision rows (T67264 T428406) (duration: 09m 54s)
  • 14:53 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2368.codfw.wmnet
  • 14:53 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host wikikube-worker1334.eqiad.wmnet
  • 14:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1334.eqiad.wmnet
  • 14:53 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker1333.eqiad.wmnet
  • 14:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1333.eqiad.wmnet
  • 14:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2368.codfw.wmnet
  • 14:53 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2367.codfw.wmnet
  • 14:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2367.codfw.wmnet
  • 14:51 dkertesz@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-upload_magru and A:cp - upgrade haproxy to 3.2.25-1 (T439667)
  • 14:49 ecarg@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
  • 14:49 zabe@deploy1003: zabe: Continuing with deployment
  • 14:49 ecarg@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
  • 14:48 ecarg@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
  • 14:48 zabe@deploy1003: zabe: Backport for Add script to drop duplicate oldimage/filerevision rows (T67264 T428406), Add script to drop duplicate oldimage/filerevision rows (T67264 T428406) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 14:48 ecarg@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
  • 14:47 ecarg@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
  • 14:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2367.codfw.wmnet
  • 14:45 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2367.codfw.wmnet
  • 14:45 oblivian@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
  • 14:45 oblivian@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'.
  • 14:45 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2366.codfw.wmnet
  • 14:45 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2366.codfw.wmnet
  • 14:45 oblivian@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'.
  • 14:45 oblivian@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'.
  • 14:44 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host wikikube-worker1333.eqiad.wmnet
  • 14:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1333.eqiad.wmnet
  • 14:44 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker1332.eqiad.wmnet
  • 14:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1332.eqiad.wmnet
  • 14:44 zabe@deploy1003: Started scap sync-world: Backport for Add script to drop duplicate oldimage/filerevision rows (T67264 T428406), Add script to drop duplicate oldimage/filerevision rows (T67264 T428406)
  • 14:44 oblivian@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
  • 14:44 oblivian@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
  • 14:43 oblivian@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'.
  • 14:43 oblivian@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'.
  • 14:43 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
  • 14:38 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2366.codfw.wmnet
  • 14:38 andrew@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudcephosd1043.eqiad.wmnet with OS bookworm
  • 14:37 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2366.codfw.wmnet
  • 14:37 ecarg@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
  • 14:37 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2365.codfw.wmnet
  • 14:37 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2365.codfw.wmnet
  • 14:37 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
  • 14:37 ecarg@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
  • 14:37 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
  • 14:36 ecarg@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
  • 14:36 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host wikikube-worker1332.eqiad.wmnet
  • 14:36 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1332.eqiad.wmnet
  • 14:36 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker1331.eqiad.wmnet
  • 14:36 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1331.eqiad.wmnet
  • 14:35 jforrester@deploy1003: Finished scap sync-world: Backport for mlwiki: Enable Wikifunctions client mode (duration: 09m 51s)
  • 14:32 ecarg@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
  • 14:31 ecarg@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
  • 14:31 ecarg@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
  • 14:30 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2365.codfw.wmnet
  • 14:30 ecarg@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
  • 14:30 jforrester@deploy1003: jforrester: Continuing with deployment
  • 14:30 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2365.codfw.wmnet
  • 14:30 ecarg@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
  • 14:29 oblivian@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on D{wikikube-worker236[5-9].codfw.wmnet} and (A:wikikube-master-codfw or A:wikikube-worker-codfw)
  • 14:29 ecarg@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
  • 14:29 jforrester@deploy1003: jforrester: Backport for mlwiki: Enable Wikifunctions client mode synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 14:27 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host wikikube-worker1331.eqiad.wmnet
  • 14:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1331.eqiad.wmnet
  • 14:27 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker1330.eqiad.wmnet
  • 14:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1330.eqiad.wmnet
  • 14:26 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply
  • 14:26 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply
  • 14:25 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2369.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 14:25 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2369.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 14:25 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply
  • 14:25 jforrester@deploy1003: Started scap sync-world: Backport for mlwiki: Enable Wikifunctions client mode
  • 14:25 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2368.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 14:25 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2368.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 14:24 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2367.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 14:24 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2367.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 14:24 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2366.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 14:24 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply
  • 14:24 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2366.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 14:24 jforrester@deploy1003: Finished scap sync-world: Backport for mlwiki: Enable Abstract Wikimedia client mode (duration: 11m 31s)
  • 14:23 ecarg@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
  • 14:22 ecarg@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
  • 14:22 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply
  • 14:22 ecarg@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
  • 14:22 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply
  • 14:21 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2365.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 14:21 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2365.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 14:21 ecarg@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
  • 14:21 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply
  • 14:21 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply
  • 14:21 ecarg@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
  • 14:20 ecarg@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
  • 14:18 jforrester@deploy1003: jforrester: Continuing with deployment
  • 14:17 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host wikikube-worker1330.eqiad.wmnet
  • 14:17 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1330.eqiad.wmnet
  • 14:17 oblivian@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on D{wikikube-worker133[0-4].eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad)
  • 14:16 ecarg@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
  • 14:16 jforrester@deploy1003: jforrester: Backport for mlwiki: Enable Abstract Wikimedia client mode synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 14:15 dkertesz@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_magru and A:cp - upgrade haproxy to 3.2.25-1 (T439667)
  • 14:15 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1334.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 14:15 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1334.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 14:15 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1333.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 14:15 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1333.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 14:14 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1332.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 14:14 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1332.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 14:14 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1331.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 14:14 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1331.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 14:14 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1330.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 14:14 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1330.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 14:14 ecarg@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
  • 14:13 ecarg@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
  • 14:12 jforrester@deploy1003: Started scap sync-world: Backport for mlwiki: Enable Abstract Wikimedia client mode
  • 14:11 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
  • 14:09 ecarg@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
  • 14:09 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
  • 14:08 ecarg@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
  • 14:07 ecarg@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
  • 14:06 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
  • 14:06 seanleong-wmde@deploy1003: Finished scap sync-world: Backport for Fix xsd xml version (T439093) (duration: 11m 33s)
  • 14:05 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
  • 14:04 Amir1: mwscript-k8s --comment="T358266" --local_dblist=<(expanddblist all | grep -v commons) -- extensions/TimedMediaHandler/maintenance/requeueTranscodes.php --key '360p.mpeg4.mov' --throttle --video --missing (T358266)
  • 14:04 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
  • 14:04 dkertesz@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-text_magru and not P{cp7001.*} and A:cp - upgrade haproxy to 3.2.25-1 (T439667)
  • 14:01 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
  • 14:00 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
  • 14:00 inflatador: bking@krb2002 disable wmf_auto_restart krb admin timers/services as this host is no longer primary T437937
  • 14:00 seanleong-wmde@deploy1003: awight, seanleong-wmde: Continuing with deployment
  • 13:58 seanleong-wmde@deploy1003: awight, seanleong-wmde: Backport for Fix xsd xml version (T439093) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 13:56 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
  • 13:54 seanleong-wmde@deploy1003: Started scap sync-world: Backport for Fix xsd xml version (T439093)
  • 13:53 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
  • 13:51 seanleong-wmde@deploy1003: Finished scap sync-world: Backport for Create new http://www.mediawiki.org/xml/sitelist-1.1/ to reference sitelist-1.1.xsd (T222516 T439093) (duration: 12m 01s)
  • 13:48 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
  • 13:46 seanleong-wmde@deploy1003: seanleong-wmde, lucamauri: Continuing with deployment
  • 13:46 andrew@cumin2003: START - Cookbook sre.hosts.reimage for host cloudcephosd1043.eqiad.wmnet with OS bookworm
  • 13:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on D{wikikube-worker237[0-5].codfw.wmnet} and (A:wikikube-master-codfw or A:wikikube-worker-codfw)
  • 13:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2374.codfw.wmnet
  • 13:46 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2374.codfw.wmnet
  • 13:43 seanleong-wmde@deploy1003: seanleong-wmde, lucamauri: Backport for Create new http://www.mediawiki.org/xml/sitelist-1.1/ to reference sitelist-1.1.xsd (T222516 T439093) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 13:41 andrew@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudcephosd1043.eqiad.wmnet with OS bookworm
  • 13:39 seanleong-wmde@deploy1003: Started scap sync-world: Backport for Create new http://www.mediawiki.org/xml/sitelist-1.1/ to reference sitelist-1.1.xsd (T222516 T439093)
  • 13:38 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2374.codfw.wmnet
  • 13:38 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2374.codfw.wmnet
  • 13:38 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2373.codfw.wmnet
  • 13:38 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2373.codfw.wmnet
  • 13:32 jgiannelos@deploy1003: Finished scap sync-world: Backport for prv: Enable parsoid rendering for 5 wikis (T439444) (duration: 11m 13s)
  • 13:32 tappof@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts kafka-logging1003.eqiad.wmnet
  • 13:32 tappof@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 13:32 tappof@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging1003.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004"
  • 13:31 tappof@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging1003.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004"
  • 13:31 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2373.codfw.wmnet
  • 13:30 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2373.codfw.wmnet
  • 13:30 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2372.codfw.wmnet
  • 13:30 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2372.codfw.wmnet
  • 13:27 jgiannelos@deploy1003: jgiannelos: Continuing with deployment
  • 13:26 tappof@cumin1004: START - Cookbook sre.dns.netbox
  • 13:25 jgiannelos@deploy1003: jgiannelos: Backport for prv: Enable parsoid rendering for 5 wikis (T439444) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 13:23 kevinbazira@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
  • 13:23 kevinbazira@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
  • 13:23 andrew@cumin2003: START - Cookbook sre.hosts.reimage for host cloudcephosd1043.eqiad.wmnet with OS bookworm
  • 13:23 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2372.codfw.wmnet
  • 13:22 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2372.codfw.wmnet
  • 13:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2371.codfw.wmnet
  • 13:22 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2371.codfw.wmnet
  • 13:21 jgiannelos@deploy1003: Started scap sync-world: Backport for prv: Enable parsoid rendering for 5 wikis (T439444)
  • 13:20 tappof@cumin1004: START - Cookbook sre.hosts.decommission for hosts kafka-logging1003.eqiad.wmnet
  • 13:18 samtar@deploy1003: Finished scap sync-world: Backport for fix(AccountSetup): add missing i18n key to extension.json (T438908 T436872) (duration: 12m 23s)
  • 13:15 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2371.codfw.wmnet
  • 13:15 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2371.codfw.wmnet
  • 13:15 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2370.codfw.wmnet
  • 13:15 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2370.codfw.wmnet
  • 13:13 samtar@deploy1003: migr, samtar: Continuing with deployment
  • 13:12 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
  • 13:10 dkertesz@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_magru and not P{cp7001.*} and A:cp - upgrade haproxy to 3.2.25-1 (T439667)
  • 13:10 samtar@deploy1003: migr, samtar: Backport for fix(AccountSetup): add missing i18n key to extension.json (T438908 T436872) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 13:07 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2370.codfw.wmnet
  • 13:07 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2370.codfw.wmnet
  • 13:07 oblivian@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on D{wikikube-worker237[0-5].codfw.wmnet} and (A:wikikube-master-codfw or A:wikikube-worker-codfw)
  • 13:06 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2374.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 13:06 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2374.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 13:06 samtar@deploy1003: Started scap sync-world: Backport for fix(AccountSetup): add missing i18n key to extension.json (T438908 T436872)
  • 13:05 tappof@deploy1003: Finished scap sync-world: T432444 - remove kafka-logging100[12] (duration: 10m 41s)
  • 13:05 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2373.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 13:05 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2373.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 13:04 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2372.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 13:04 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2372.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 13:04 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2371.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 13:04 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2371.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 13:04 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2370.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 13:04 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2370.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
  • 13:02 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
  • 12:56 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
  • 12:56 tappof@deploy1003: Started scap sync-world: T432444 - remove kafka-logging100[12]
  • 12:54 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
  • 12:54 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
  • 12:54 tappof@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'sync'.
  • 12:53 tappof@deploy1003: helmfile [codfw] START helmfile.d/admin 'sync'.
  • 12:53 tappof@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'sync'.
  • 12:53 tappof@deploy1003: helmfile [eqiad] START helmfile.d/admin 'sync'.
  • 12:48 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
  • 12:47 dkertesz@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on P{cp7001.*} and A:cp - upgrade haproxy to 3.2.25-1 (T439667)
  • 12:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on D{wikikube-worker138[0-3].eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad)
  • 12:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1383.eqiad.wmnet
  • 12:46 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1383.eqiad.wmnet
  • 12:45 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
  • 12:44 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
  • 12:43 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
  • 12:43 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
  • 12:43 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
  • 12:42 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
  • 12:41 dkertesz@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on P{cp7001.*} and A:cp - upgrade haproxy to 3.2.25-1 (T439667)
  • 12:39 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1383.eqiad.wmnet
  • 12:39 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1383.eqiad.wmnet
  • 12:39 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1382.eqiad.wmnet
  • 12:39 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1382.eqiad.wmnet
  • 12:37 krinkle@deploy1003: Finished scap sync-world: Backport for Profiler: Fix excimer component regex skipping the last frame in each stack (duration: 10m 56s)
  • 12:33 dkertesz: fetching new haproxy packages on reprepro (3.2.23-1~bpo13+1 -> 3.2.25-1~bpo13+1) - T439667
  • 12:32 krinkle@deploy1003: somerandomdeveloper, krinkle: Continuing with deployment
  • 12:32 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1382.eqiad.wmnet
  • 12:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1382.eqiad.wmnet
  • 12:31 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1381.eqiad.wmnet
  • 12:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1381.eqiad.wmnet
  • 12:30 krinkle@deploy1003: somerandomdeveloper, krinkle: Backport for Profiler: Fix excimer component regex skipping the last frame in each stack synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 12:26 krinkle@deploy1003: Started scap sync-world: Backport for Profiler: Fix excimer component regex skipping the last frame in each stack
  • 12:24 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1381.eqiad.wmnet
  • 12:23 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1381.eqiad.wmnet
  • 12:23 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1380.eqiad.wmnet
  • 12:23 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1380.eqiad.wmnet
  • 12:16 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1380.eqiad.wmnet
  • 12:16 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1380.eqiad.wmnet
  • 12:16 oblivian@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on D{wikikube-worker138[0-3].eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad)
  • 12:15 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1383.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 12:14 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1383.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 12:14 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1382.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 12:14 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1382.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 12:14 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1381.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 12:14 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1381.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 12:14 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1380.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 12:13 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1380.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 12:12 tappof@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5 days, 0:00:00 on kafka-logging[1001-1003].eqiad.wmnet with reason: stop service on kafka-logging100[1-3]
  • 12:02 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1043.eqiad.wmnet
  • 12:00 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1044.eqiad.wmnet
  • 11:55 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1044.eqiad.wmnet
  • 11:55 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1043.eqiad.wmnet
  • 11:41 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
  • 11:39 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1042.eqiad.wmnet
  • 11:36 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
  • 11:34 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1042.eqiad.wmnet
  • 11:33 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1043.eqiad.wmnet
  • 11:33 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1044.eqiad.wmnet
  • 11:32 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1044.eqiad.wmnet
  • 11:32 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1043.eqiad.wmnet
  • 11:32 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1042.eqiad.wmnet
  • 11:32 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1042.eqiad.wmnet
  • 11:29 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
  • 11:27 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
  • 11:25 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply
  • 11:24 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply
  • 11:24 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
  • 11:24 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply
  • 11:24 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
  • 11:24 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
  • 11:24 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply
  • 11:23 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
  • 11:22 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-jobrunner: apply
  • 11:22 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/mw-jobrunner: apply
  • 11:22 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-jobrunner: apply
  • 11:22 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-jobrunner: apply
  • 11:07 cgoubert@deploy1003: Finished scap sync-world: 1330505: mediawiki: Redirect /api/ to /w/rest.php | T433547 (duration: 11m 12s)
  • 11:01 cgoubert@deploy1003: cgoubert: Continuing with deployment
  • 10:59 cgoubert@deploy1003: cgoubert: 1330505: mediawiki: Redirect /api/ to /w/rest.php | T433547 synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 10:56 cgoubert@deploy1003: Started scap sync-world: 1330505: mediawiki: Redirect /api/ to /w/rest.php | T433547
  • 10:40 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on D{wikikube-worker1384.eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad)
  • 10:40 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1384.eqiad.wmnet
  • 10:40 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1384.eqiad.wmnet
  • 10:38 elukey: `sudo ipvsadm --delete-service --tcp-service 10.2.1.96:443` on lvs201[34] to remove the pki service
  • 10:33 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1384.eqiad.wmnet
  • 10:33 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1384.eqiad.wmnet
  • 10:33 oblivian@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on D{wikikube-worker1384.eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad)
  • 10:32 tappof: Grafana upgrade from 13.2.0 to 13.2.3 - T438454
  • 10:27 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1384.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 10:27 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1384.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 10:06 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-eqiad@eqiad
  • 10:06 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs
  • 10:05 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs
  • 10:03 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad
  • 09:39 jelto@cumin1004: END (FAIL) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=99) for alias: wikikube-worker-eqiad@eqiad
  • 09:38 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad
  • 09:33 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-codfw@codfw
  • 09:33 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs
  • 09:32 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs
  • 09:28 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-codfw@codfw
  • 09:08 btullis@dns1004: END - running authdns-update
  • 09:06 btullis@dns1004: START - running authdns-update
  • 09:04 elukey: elukey@cumin1004:~$ sudo cumin 'A:lvs-low-traffic-codfw' 'systemctl restart pybal.service'
  • 09:01 elukey: elukey@cumin1004:~$ sudo cumin 'A:lvs-secondary-codfw' 'systemctl restart pybal.service'
  • 08:27 jmm@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe-eqiad
  • 08:25 jmm@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe-eqiad
  • 08:24 elukey@cumin1004: conftool action : set/pooled=yes:weight=1; selector: cluster=pki,service=cfssl-multirootca
  • 08:24 jmm@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe-codfw
  • 08:22 jmm@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe-codfw
  • 07:49 filippo@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudcephosd1056.eqiad.wmnet with OS bookworm
  • 07:46 elukey@dns1004: END - running authdns-update
  • 07:44 kevinbazira@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
  • 07:44 kevinbazira@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
  • 07:44 elukey@dns1004: START - running authdns-update
  • 07:35 moritzm: installing pcre2/openssl security updates
  • 07:18 filippo@cumin1004: START - Cookbook sre.hosts.reimage for host cloudcephosd1056.eqiad.wmnet with OS bookworm
  • 07:18 filippo@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudcephosd1056.eqiad.wmnet with OS bookworm
  • 07:15 dcausse@deploy1003: Finished scap sync-world: Backport for Search: optimize morelike queries (duration: 11m 27s)
  • 07:13 filippo@cumin1004: START - Cookbook sre.hosts.reimage for host cloudcephosd1056.eqiad.wmnet with OS bookworm
  • 07:12 filippo@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudcephosd1056.eqiad.wmnet with OS bookworm
  • 07:09 dcausse@deploy1003: dcausse: Continuing with deployment
  • 07:07 dcausse@deploy1003: dcausse: Backport for Search: optimize morelike queries synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 07:03 dcausse@deploy1003: Started scap sync-world: Backport for Search: optimize morelike queries
  • 06:51 filippo@cumin1004: START - Cookbook sre.hosts.reimage for host cloudcephosd1056.eqiad.wmnet with OS bookworm
  • 06:48 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host cloudcephosd1056.eqiad.wmnet
  • 06:33 filippo@cumin1004: START - Cookbook sre.hosts.dhcp for host cloudcephosd1056.eqiad.wmnet
  • 06:32 filippo@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudcephosd1055.eqiad.wmnet with OS bookworm
  • 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 40s)
  • 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image

2026-09-29

  • 22:42 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1044.eqiad.wmnet with OS bookworm
  • 22:39 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1043.eqiad.wmnet with OS bookworm
  • 22:26 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1044.eqiad.wmnet with reason: host reimage
  • 22:23 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1043.eqiad.wmnet with reason: host reimage
  • 22:21 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1044.eqiad.wmnet with reason: host reimage
  • 22:19 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1043.eqiad.wmnet with reason: host reimage
  • 22:07 bking@cumin2003: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1044.eqiad.wmnet with OS bookworm
  • 22:06 bking@cumin2003: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ml-cache1003 to dse-k8s-worker1044
  • 22:06 bking@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1044
  • 22:05 bking@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1044
  • 22:05 bking@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1044 on all recursors
  • 22:05 bking@cumin2003: START - Cookbook sre.dns.wipe-cache dse-k8s-worker1044 on all recursors
  • 22:05 bking@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 22:05 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ml-cache1003 to dse-k8s-worker1044 - bking@cumin2003"
  • 22:05 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ml-cache1003 to dse-k8s-worker1044 - bking@cumin2003"
  • 22:03 bking@cumin2003: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1043.eqiad.wmnet with OS bookworm
  • 22:03 bking@cumin2003: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ml-cache1002 to dse-k8s-worker1043
  • 22:02 bking@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1043
  • 22:01 bking@cumin2003: START - Cookbook sre.dns.netbox
  • 22:01 bking@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1043
  • 22:01 bking@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1043 on all recursors
  • 22:01 bking@cumin2003: START - Cookbook sre.dns.wipe-cache dse-k8s-worker1043 on all recursors
  • 22:01 bking@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 22:01 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ml-cache1002 to dse-k8s-worker1043 - bking@cumin2003"
  • 22:01 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ml-cache1002 to dse-k8s-worker1043 - bking@cumin2003"
  • 21:56 bking@cumin2003: START - Cookbook sre.hosts.rename from ml-cache1003 to dse-k8s-worker1044
  • 21:56 bking@cumin2003: START - Cookbook sre.dns.netbox
  • 21:56 bking@cumin2003: START - Cookbook sre.hosts.rename from ml-cache1002 to dse-k8s-worker1043
  • 21:56 bking@cumin2003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for 1 hosts
  • 21:55 bking@cumin2003: START - Cookbook sre.hosts.bmc-user-mgmt for 1 hosts
  • 21:55 bking@cumin2003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for 1 hosts
  • 21:55 bking@cumin2003: START - Cookbook sre.hosts.bmc-user-mgmt for 1 hosts
  • 21:29 mutante: decom'ing VM zuul1002 - replaced by physical machine zuul1005 - T427353 T395938
  • 21:27 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1042.eqiad.wmnet with OS bookworm
  • 21:20 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply
  • 21:20 eevans@deploy1003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply
  • 21:16 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply
  • 21:16 eevans@deploy1003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply
  • 21:11 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1042.eqiad.wmnet with reason: host reimage
  • 21:08 cklimas@deploy1003: Finished scap sync-world: Backport for Beta Cluster: Use different Redis instance for LockManager (T436480) (duration: 12m 45s)
  • 21:06 bking@cumin2003: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['ml-cache1003.eqiad.wmnet']
  • 21:06 bking@cumin2003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ml-cache1003.eqiad.wmnet']
  • 21:06 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1042.eqiad.wmnet with reason: host reimage
  • 21:04 bking@cumin2003: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['ml-cache1002.eqiad.wmnet']
  • 21:03 bking@cumin2003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ml-cache1002.eqiad.wmnet']
  • 21:03 bking@cumin2003: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['ml-cache1002.eqiad.wmnet']
  • 21:03 bking@cumin2003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ml-cache1002.eqiad.wmnet']
  • 21:02 cklimas@deploy1003: cklimas, southparkfan: Continuing with deployment
  • 20:59 cklimas@deploy1003: cklimas, southparkfan: Backport for Beta Cluster: Use different Redis instance for LockManager (T436480) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 20:55 cklimas@deploy1003: Started scap sync-world: Backport for Beta Cluster: Use different Redis instance for LockManager (T436480)
  • 20:51 bking@cumin2003: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1042.eqiad.wmnet with OS bookworm
  • 20:48 cklimas@deploy1003: Finished scap sync-world: Backport for InitialiseSettings: Enable return-to-app banner on testwiki (T434258) (duration: 14m 32s)
  • 20:47 bking@cumin2003: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ml-cache1001 to dse-k8s-worker1042
  • 20:46 bking@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1042
  • 20:45 bking@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1042
  • 20:45 bking@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1042 on all recursors
  • 20:45 bking@cumin2003: START - Cookbook sre.dns.wipe-cache dse-k8s-worker1042 on all recursors
  • 20:45 bking@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 20:45 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ml-cache1001 to dse-k8s-worker1042 - bking@cumin2003"
  • 20:43 cklimas@deploy1003: cklimas: Continuing with deployment
  • 20:42 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ml-cache1001 to dse-k8s-worker1042 - bking@cumin2003"
  • 20:38 cklimas@deploy1003: cklimas: Backport for InitialiseSettings: Enable return-to-app banner on testwiki (T434258) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 20:33 cklimas@deploy1003: Started scap sync-world: Backport for InitialiseSettings: Enable return-to-app banner on testwiki (T434258)
  • 20:33 bking@cumin2003: START - Cookbook sre.dns.netbox
  • 20:32 bking@cumin2003: START - Cookbook sre.hosts.rename from ml-cache1001 to dse-k8s-worker1042
  • 20:22 sbassett@deploy1003: Finished scap sync-world: Backport for Content Security Policy: add stricter report-only policy (T419612 T427557 T431144 T327588 T245424), Remove foundationwiki's CSP report-only header (T423691) (duration: 14m 07s)
  • 20:19 sbassett@deploy1003: sbassett: Rolling back deployment
  • 20:12 sbassett@deploy1003: sbassett: Backport for Content Security Policy: add stricter report-only policy (T419612 T427557 T431144 T327588 T245424), Remove foundationwiki's CSP report-only header (T423691) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 20:12 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-cache1002.eqiad.wmnet with OS bookworm
  • 20:11 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004"
  • 20:09 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004"
  • 20:08 sbassett@deploy1003: Started scap sync-world: Backport for Content Security Policy: add stricter report-only policy (T419612 T427557 T431144 T327588 T245424), Remove foundationwiki's CSP report-only header (T423691)
  • 20:04 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-cache1001.eqiad.wmnet with OS bookworm
  • 20:04 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004"
  • 19:57 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
  • 19:56 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply
  • 19:56 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
  • 19:56 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply
  • 19:55 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004"
  • 19:55 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-cache1003.eqiad.wmnet with OS bookworm
  • 19:55 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004"
  • 19:53 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-cache1002.eqiad.wmnet with reason: host reimage
  • 19:53 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-text_eqsin - 7.1.1-2~bpo13+wmf3 ()
  • 19:52 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004"
  • 19:48 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ml-cache1002.eqiad.wmnet with reason: host reimage
  • 19:40 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-cache1001.eqiad.wmnet with reason: host reimage
  • 19:40 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply
  • 19:39 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply
  • 19:38 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ml-cache1001.eqiad.wmnet with reason: host reimage
  • 19:37 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-cache1003.eqiad.wmnet with reason: host reimage
  • 19:33 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host ml-cache1002.eqiad.wmnet with OS bookworm
  • 19:33 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ml-cache1003.eqiad.wmnet with reason: host reimage
  • 19:30 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply
  • 19:29 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply
  • 19:27 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ml-cache1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
  • 19:24 kemayo@deploy1003: helmfile [staging] DONE helmfile.d/services/editcheck-headless: apply
  • 19:23 kemayo@deploy1003: helmfile [staging] START helmfile.d/services/editcheck-headless: apply
  • 19:22 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host ml-cache1001.eqiad.wmnet with OS bookworm
  • 19:19 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host ml-cache1003.eqiad.wmnet with OS bookworm
  • 19:17 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-cache1003.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
  • 19:15 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
  • 19:15 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
  • 19:10 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ml-cache1001.eqiad.wmnet with OS bookworm
  • 19:10 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host ml-cache1001.eqiad.wmnet with OS bookworm
  • 19:09 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
  • 19:09 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply
  • 19:09 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply
  • 19:08 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache1003.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
  • 19:08 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
  • 19:08 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
  • 19:08 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ml-cache1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
  • 19:07 btullis@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host ml-cache1003.eqiad.wmnet with OS bookworm
  • 19:06 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
  • 19:06 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
  • 19:05 btullis@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host ml-cache1002.eqiad.wmnet with OS bookworm
  • 19:05 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
  • 19:04 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host ml-cache1003.eqiad.wmnet with OS bookworm
  • 19:03 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ml-cache1001.eqiad.wmnet with OS bookworm
  • 19:03 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host ml-cache1001.eqiad.wmnet with OS bookworm
  • 19:02 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ml-cache1001.eqiad.wmnet ml-cache1003.eqiad.wmnet on all recursors
  • 19:02 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache ml-cache1001.eqiad.wmnet ml-cache1003.eqiad.wmnet on all recursors
  • 18:57 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir5004.*
  • 18:56 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 18:56 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache servers - btullis@cumin1004"
  • 18:56 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache servers - btullis@cumin1004"
  • 18:56 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host ml-cache1002.eqiad.wmnet with OS bookworm
  • 18:50 btullis@cumin1004: START - Cookbook sre.dns.netbox
  • 18:49 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 18:49 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache1001 network - btullis@cumin1004"
  • 18:49 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache1001 network - btullis@cumin1004"
  • 18:45 btullis@cumin1004: START - Cookbook sre.dns.netbox
  • 18:41 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 18:41 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache1001 network - btullis@cumin1004"
  • 18:39 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache1001 network - btullis@cumin1004"
  • 18:35 btullis@cumin1004: START - Cookbook sre.dns.netbox
  • 18:34 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 18:34 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache1002 and ml-cache1003 - btullis@cumin1004"
  • 18:31 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache1002 and ml-cache1003 - btullis@cumin1004"
  • 18:30 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply
  • 18:30 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply
  • 18:27 btullis@cumin1004: START - Cookbook sre.dns.netbox
  • 18:26 ryankemper@cumin2003: conftool action : set/weight=10:pooled=yes; selector: name=wdqs1033.eqiad.wmnet
  • 18:26 ryankemper@cumin2003: conftool action : set/weight=10:pooled=yes; selector: name=wdqs1034.eqiad.wmnet
  • 18:26 ryankemper@cumin2003: conftool action : set/weight=10:pooled=yes; selector: name=wdqs1035.eqiad.wmnet
  • 18:25 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
  • 18:24 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
  • 18:19 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply
  • 18:19 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply
  • 18:19 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply
  • 18:18 brennen@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.47.0-wmf.22 refs T438218
  • 18:18 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply
  • 18:15 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply
  • 18:15 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply
  • 18:15 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply
  • 18:08 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T420707, T420707: initial data for new host) xfer wdqs-all from wdqs2021.codfw.wmnet -> wdqs1033.eqiad.wmnet, repooling source-only afterwards
  • 18:06 brennen: 1.47.0-wmf.22 train status (T438218) - no current blockers, rolling to group0
  • 18:05 sandraebele: Deployed Refinery as part of weekly deployment train
  • 18:05 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-upload_eqsin - 7.1.1-2~bpo13+wmf3 ()
  • 18:01 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T420707, T420707: initial data for new host) xfer wdqs-all from wdqs2013.codfw.wmnet -> wdqs1034.eqiad.wmnet, repooling source-only afterwards
  • 17:59 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T420707, T420707: initial data for new host) xfer scholarly_articles from wdqs2023.codfw.wmnet -> wdqs1035.eqiad.wmnet, repooling source-only afterwards
  • 17:58 ebysans@deploy1003: Finished deploy [analytics/refinery@613ef4d]: Regular analytics weekly train [analytics/refinery@613ef4d9] (duration: 04m 54s)
  • 17:57 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply
  • 17:55 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply
  • 17:54 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply
  • 17:54 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply
  • 17:53 ebysans@deploy1003: Started deploy [analytics/refinery@613ef4d]: Regular analytics weekly train [analytics/refinery@613ef4d9]
  • 17:53 sandraebele: Deploying Refinery as part of weekly deployment train
  • 17:52 ebysans@deploy1003: Finished deploy [analytics/refinery@613ef4d] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@613ef4d9] (duration: 00m 46s)
  • 17:51 ebysans@deploy1003: Started deploy [analytics/refinery@613ef4d] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@613ef4d9]
  • 17:51 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir5004.eqsin.wmnet with OS trixie
  • 17:50 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply
  • 17:46 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
  • 17:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
  • 17:44 sandraebele: Deployed refinery-source using jenkins as part of weekly deployment train
  • 17:39 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
  • 17:01 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer (T420707, T420707: initial data for new host) xfer scholarly_articles from wdqs2023.codfw.wmnet -> wdqs1035.eqiad.wmnet, repooling source-only afterwards
  • 17:01 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer (T420707, T420707: initial data for new host) xfer wdqs-all from wdqs2013.codfw.wmnet -> wdqs1034.eqiad.wmnet, repooling source-only afterwards
  • 17:00 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer (T420707, T420707: initial data for new host) xfer wdqs-all from wdqs2021.codfw.wmnet -> wdqs1033.eqiad.wmnet, repooling source-only afterwards
  • 16:58 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp5028.*
  • 16:54 ryankemper@deploy1003: Finished deploy [wdqs/wdqs@cf64fe9]: T420707: deploy to fresh wdqs host (duration: 00m 04s)
  • 16:54 ryankemper@deploy1003: Started deploy [wdqs/wdqs@cf64fe9]: T420707: deploy to fresh wdqs host
  • 16:54 ryankemper@deploy1003: Finished deploy [wdqs/wdqs@cf64fe9]: T420707: deploy to fresh wdqs host (duration: 00m 04s)
  • 16:54 ryankemper@deploy1003: Started deploy [wdqs/wdqs@cf64fe9]: T420707: deploy to fresh wdqs host
  • 16:54 ryankemper@deploy1003: Finished deploy [wdqs/wdqs@cf64fe9]: T420707: deploy to fresh wdqs host (duration: 00m 04s)
  • 16:54 ryankemper@deploy1003: Started deploy [wdqs/wdqs@cf64fe9]: T420707: deploy to fresh wdqs host
  • 16:36 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: name=cp5028.eqsin.wmnet
  • 16:36 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: name=cp5028
  • 16:35 inflatador: bking@cephosd1001 `sudo radosgw-admin user create --uid=matomo --display-name="Matomo on K8s"` T436186
  • 16:22 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
  • 16:21 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
  • 16:20 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
  • 16:19 urbanecm@deploy1003: mwscript-k8s job started: GrowthExperiments:reassignMentees --wiki=enwiki --mentor=Blacephalon --performer=Blacephalon --as-job # T418194
  • 16:19 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
  • 16:19 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
  • 16:19 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
  • 16:18 jhathaway@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-cron: apply
  • 16:16 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
  • 16:15 jhathaway@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-cron: apply
  • 15:59 jhathaway: updating puppet stdlib, should be a noop
  • 15:43 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-upload_eqsin - 7.1.1-2~bpo13+wmf3 ()
  • 15:39 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-text_eqsin - 7.1.1-2~bpo13+wmf3 ()
  • 15:23 dzahn@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host zuul1005.eqiad.wmnet with OS trixie
  • 15:14 cdobbins@cumin1004: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on ncredir5004.eqsin.wmnet with reason: host reimage
  • 15:09 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir5004.eqsin.wmnet with reason: host reimage
  • 15:08 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
  • 15:08 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
  • 14:32 cgoubert@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on wikikube-worker1152.eqiad.wmnet with reason: hardware/networking issues
  • 14:23 cmooney@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
  • 14:21 cmooney@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
  • 14:21 moritzm: installing rsyslog security updates
  • 14:19 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir5004.eqsin.wmnet with OS trixie
  • 14:16 klausman@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for ml-serve1015.eqiad.wmnet
  • 14:16 klausman@cumin1004: START - Cookbook sre.hosts.remove-downtime for ml-serve1015.eqiad.wmnet
  • 14:12 bking@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=k8s-ingress-dse-aa,name=codfw
  • 14:12 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
  • 14:11 krinkle@deploy1003: Finished scap sync-world: Backport for File: Implement LocalFile::getUrlForPurge workaround for caching (T425216), ProductionServices: Simplify indirect parsercache-dbs stub (T362786 T437209) (duration: 14m 26s)
  • 14:07 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
  • 14:07 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 14:07 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache1001 - btullis@cumin1004"
  • 14:06 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache1001 - btullis@cumin1004"
  • 14:05 krinkle@deploy1003: krinkle: Continuing with deployment
  • 14:02 btullis@cumin1004: START - Cookbook sre.dns.netbox
  • 14:00 krinkle@deploy1003: krinkle: Backport for File: Implement LocalFile::getUrlForPurge workaround for caching (T425216), ProductionServices: Simplify indirect parsercache-dbs stub (T362786 T437209) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 13:59 cmooney@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary
  • 13:58 cmooney@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
  • 13:56 krinkle@deploy1003: Started scap sync-world: Backport for File: Implement LocalFile::getUrlForPurge workaround for caching (T425216), ProductionServices: Simplify indirect parsercache-dbs stub (T362786 T437209)
  • 13:54 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
  • 13:51 krinkle@deploy1003: Finished deploy [performance/arc-lamp@35c590e]: T331941 (duration: 00m 09s)
  • 13:51 krinkle@deploy1003: Started deploy [performance/arc-lamp@35c590e]: T331941
  • 13:47 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host testvm2001.codfw.wmnet
  • 13:47 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host testvm2001.codfw.wmnet with OS trixie
  • 13:47 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
  • 13:46 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for fix: prevent AccountSetup from being scrollable (T436872), fix: prevent AccountSetup from being scrollable (T436872) (duration: 11m 39s)
  • 13:40 lucaswerkmeister-wmde@deploy1003: migr, lucaswerkmeister-wmde: Continuing with deployment
  • 13:38 lucaswerkmeister-wmde@deploy1003: migr, lucaswerkmeister-wmde: Backport for fix: prevent AccountSetup from being scrollable (T436872), fix: prevent AccountSetup from being scrollable (T436872) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 13:34 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for fix: prevent AccountSetup from being scrollable (T436872), fix: prevent AccountSetup from being scrollable (T436872)
  • 13:30 ihurbain@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
  • 13:30 ihurbain@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
  • 13:25 samtar@deploy1003: Finished scap sync-world: Backport for CommonSettings: Load ModeratorToolkit extension (T438334) (duration: 11m 17s)
  • 13:21 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
  • 13:20 samtar@deploy1003: samtar: Continuing with deployment
  • 13:19 samtar@deploy1003: samtar: Backport for CommonSettings: Load ModeratorToolkit extension (T438334) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 13:16 ihurbain@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
  • 13:16 ihurbain@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
  • 13:15 ihurbain@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
  • 13:15 ihurbain@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
  • 13:15 ihurbain@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
  • 13:15 ihurbain@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
  • 13:14 samtar@deploy1003: Started scap sync-world: Backport for CommonSettings: Load ModeratorToolkit extension (T438334)
  • 13:10 samtar@deploy1003: Finished scap sync-world: Backport for InitialiseSettings: Enable ModeratorToolkit on test.wiki (T438334) (duration: 10m 21s)
  • 13:05 samtar@deploy1003: samtar: Continuing with deployment
  • 13:04 samtar@deploy1003: samtar: Backport for InitialiseSettings: Enable ModeratorToolkit on test.wiki (T438334) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 13:00 samtar@deploy1003: Started scap sync-world: Backport for InitialiseSettings: Enable ModeratorToolkit on test.wiki (T438334)
  • 12:54 ihurbain@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
  • 12:54 ihurbain@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
  • 12:53 klausman@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on ml-serve1015.eqiad.wmnet with reason: Downtime for more BIOS setup rummaging
  • 12:52 urbanecm@deploy1003: Finished scap sync-world: Backport for ReassignMentees: Make the jobReleaseTimestamp work (T418194) (duration: 12m 26s)
  • 12:47 urbanecm@deploy1003: urbanecm: Continuing with deployment
  • 12:44 urbanecm@deploy1003: urbanecm: Backport for ReassignMentees: Make the jobReleaseTimestamp work (T418194) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 12:40 urbanecm@deploy1003: Started scap sync-world: Backport for ReassignMentees: Make the jobReleaseTimestamp work (T418194)
  • 12:39 filippo@cumin1004: START - Cookbook sre.hosts.reimage for host cloudcephosd1055.eqiad.wmnet with OS bookworm
  • 12:31 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
  • 12:30 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
  • 12:28 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
  • 12:27 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
  • 12:06 osleger@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
  • 12:06 osleger@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
  • 12:06 osleger@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
  • 12:06 osleger@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
  • 12:03 osleger@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
  • 12:03 osleger@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
  • 12:03 osleger@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
  • 12:02 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
  • 12:02 osleger@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
  • 10:56 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
  • 10:56 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
  • 10:56 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
  • 10:55 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
  • 10:55 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
  • 10:47 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
  • 10:46 tappof: Rebalancing has been completed on kafka-logging in eqiad: kafka-logging1001 and kafka-logging1002 are no longer assigned any partitions — T432444
  • 10:43 atsukoito: shutting down https://airflow-test-k8s.wikimedia.org for upgrade, https://phabricator.wikimedia.org/T433383#12363530
  • 10:38 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on testvm2001.codfw.wmnet with reason: host reimage
  • 10:37 blake@deploy1003: Finished scap sync-world: add sidecar flag for T417800 (duration: 04m 14s)
  • 10:33 blake@deploy1003: Started scap sync-world: add sidecar flag for T417800
  • 10:32 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on testvm2001.codfw.wmnet with reason: host reimage
  • 10:19 claime: deploy2002:~$ sudo lvextend -L +50GB --resizefs /dev/vg0/srv # T438232
  • 10:17 claime: deploy1003:~$ sudo lvextend -L +50GB --resizefs /dev/vg0/srv # T438232
  • 10:17 claime: deploy2003:~$ sudo lvextend -L +50GB --resizefs /dev/vg0/srv # T438232
  • 10:15 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host testvm2001.codfw.wmnet with OS trixie
  • 10:15 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM testvm2001.codfw.wmnet - jmm@cumin2003"
  • 10:14 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM testvm2001.codfw.wmnet - jmm@cumin2003"
  • 10:14 jmm@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) testvm2001.codfw.wmnet on all recursors
  • 10:14 jmm@cumin2003: START - Cookbook sre.dns.wipe-cache testvm2001.codfw.wmnet on all recursors
  • 10:14 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 10:14 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM testvm2001.codfw.wmnet - jmm@cumin2003"
  • 10:14 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM testvm2001.codfw.wmnet - jmm@cumin2003"
  • 10:10 ihurbain@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
  • 10:10 jmm@cumin2003: START - Cookbook sre.dns.netbox
  • 10:10 jmm@cumin2003: START - Cookbook sre.ganeti.makevm for new host testvm2001.codfw.wmnet
  • 10:09 ihurbain@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
  • 09:36 urbanecm@deploy1003: mwscript-k8s job started: GrowthExperiments:reassignMentees --wiki=enwiki --mentor=Ikan --performer=Ikan --as-job # T418194
  • 08:59 urbanecm@deploy1003: Finished scap sync-world: Backport for ReassignMentees: Actually deduplicate reassignMenteesJob (T418194) (duration: 11m 07s)
  • 08:54 urbanecm@deploy1003: urbanecm: Continuing with deployment
  • 08:52 urbanecm@deploy1003: urbanecm: Backport for ReassignMentees: Actually deduplicate reassignMenteesJob (T418194) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 08:50 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync
  • 08:49 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync
  • 08:48 urbanecm@deploy1003: Started scap sync-world: Backport for ReassignMentees: Actually deduplicate reassignMenteesJob (T418194)
  • 08:42 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-e5-eqiad
  • 08:41 ayounsi@cumin1004: START - Cookbook sre.network.tls for network device lsw1-e5-eqiad
  • 08:41 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-f5-eqiad
  • 08:41 ayounsi@cumin1004: START - Cookbook sre.network.tls for network device lsw1-f5-eqiad
  • 08:41 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-f6-eqiad
  • 08:41 ayounsi@cumin1004: START - Cookbook sre.network.tls for network device lsw1-f6-eqiad
  • 08:41 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-e7-eqiad
  • 08:40 ayounsi@cumin1004: START - Cookbook sre.network.tls for network device lsw1-e7-eqiad
  • 08:40 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-e6-eqiad
  • 08:40 ayounsi@cumin1004: START - Cookbook sre.network.tls for network device lsw1-e6-eqiad
  • 08:39 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-f7-eqiad
  • 08:39 ayounsi@cumin1004: START - Cookbook sre.network.tls for network device lsw1-f7-eqiad
  • 08:18 isaranto@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
  • 08:10 moritzm: installing PHP 8.4 security updates
  • 07:49 matthiasmullie: UTC morning backport window done
  • 07:49 mlitn@deploy1003: Finished scap sync-world: Backport for carousel: keep the hide button attached to the carousel (T439507), carousel: keep the hide button attached to the carousel (T439507) (duration: 09m 57s)
  • 07:44 mlitn@deploy1003: mlitn: Continuing with deployment
  • 07:43 mlitn@deploy1003: mlitn: Backport for carousel: keep the hide button attached to the carousel (T439507), carousel: keep the hide button attached to the carousel (T439507) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 07:39 mlitn@deploy1003: Started scap sync-world: Backport for carousel: keep the hide button attached to the carousel (T439507), carousel: keep the hide button attached to the carousel (T439507)
  • 07:35 kharlan@deploy1003: Finished scap sync-world: Backport for CheckUser: Enable ClientHintsInEditRequest on group0 (T401808) (duration: 12m 41s)
  • 07:33 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sretest2004.codfw.wmnet
  • 07:30 kharlan@deploy1003: kharlan: Continuing with deployment
  • 07:27 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host sretest2004.codfw.wmnet
  • 07:27 kharlan@deploy1003: kharlan: Backport for CheckUser: Enable ClientHintsInEditRequest on group0 (T401808) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 07:26 kart_: Updated cxserver to 2026-09-25-114231-production
  • 07:24 kartik@deploy1003: helmfile [eqiad] DONE helmfile.d/services/cxserver: apply
  • 07:24 kartik@deploy1003: helmfile [eqiad] START helmfile.d/services/cxserver: apply
  • 07:23 kartik@deploy1003: helmfile [codfw] DONE helmfile.d/services/cxserver: apply
  • 07:23 kharlan@deploy1003: Started scap sync-world: Backport for CheckUser: Enable ClientHintsInEditRequest on group0 (T401808)
  • 07:22 kartik@deploy1003: helmfile [codfw] START helmfile.d/services/cxserver: apply
  • 07:18 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host idp-test2005.wikimedia.org
  • 07:18 mlitn@deploy1003: Finished scap sync-world: Backport for Skip expensive work if known to be useless (T439182), Cache extracted carousel data (T439182), Make carousel images are visible in the margins (T439422), carousel: fix carousel image preload observer root (T439422) (duration: 13m 39s)
  • 07:14 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host idp-test2005.wikimedia.org
  • 07:11 mlitn@deploy1003: mlitn: Continuing with deployment
  • 07:09 mlitn@deploy1003: mlitn: Backport for Skip expensive work if known to be useless (T439182), Cache extracted carousel data (T439182), Make carousel images are visible in the margins (T439422), carousel: fix carousel image preload observer root (T439422) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now
  • 07:04 mlitn@deploy1003: Started scap sync-world: Backport for Skip expensive work if known to be useless (T439182), Cache extracted carousel data (T439182), Make carousel images are visible in the margins (T439422), carousel: fix carousel image preload observer root (T439422)
  • 07:02 kartik@deploy1003: helmfile [staging] DONE helmfile.d/services/cxserver: apply
  • 07:01 kartik@deploy1003: helmfile [staging] START helmfile.d/services/cxserver: apply
  • 06:57 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/etherpad: apply
  • 06:56 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/etherpad: apply
  • 05:54 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs1035.eqiad.wmnet with OS bookworm
  • 05:50 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs1034.eqiad.wmnet with OS bookworm
  • 05:38 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs1035.eqiad.wmnet with reason: host reimage
  • 05:34 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs1034.eqiad.wmnet with reason: host reimage
  • 05:32 ryankemper@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs1035.eqiad.wmnet with reason: host reimage
  • 05:31 ryankemper@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs1034.eqiad.wmnet with reason: host reimage
  • 05:21 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wdqs1035
  • 05:21 ryankemper@cumin2003: START - Cookbook sre.hosts.move-vlan for host wdqs1035
  • 05:21 ryankemper@cumin2003: START - Cookbook sre.hosts.reimage for host wdqs1035.eqiad.wmnet with OS bookworm
  • 05:19 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wdqs1034
  • 05:19 ryankemper@cumin2003: START - Cookbook sre.hosts.move-vlan for host wdqs1034
  • 05:19 ryankemper@cumin2003: START - Cookbook sre.hosts.reimage for host wdqs1034.eqiad.wmnet with OS bookworm
  • 05:03 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs1033.eqiad.wmnet with OS bookworm
  • 04:51 fabfur: manually restarting dump_cloud_ip_range on puppetserver1001 after failure
  • 04:47 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs1033.eqiad.wmnet with reason: host reimage
  • 04:43 ryankemper@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs1033.eqiad.wmnet with reason: host reimage
  • 04:32 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wdqs1033
  • 04:32 ryankemper@cumin2003: START - Cookbook sre.hosts.move-vlan for host wdqs1033
  • 04:31 ryankemper@cumin2003: START - Cookbook sre.hosts.reimage for host wdqs1033.eqiad.wmnet with OS bookworm
  • 04:01 mwpresync@deploy1003: Pruned MediaWiki: 1.47.0-wmf.19 (duration: 01m 00s)
  • 03:39 mwpresync@deploy1003: Finished scap sync-world: testwikis to 1.47.0-wmf.22 refs T438218 (duration: 36m 03s)
  • 03:03 mwpresync@deploy1003: Started scap sync-world: testwikis to 1.47.0-wmf.22 refs T438218
  • 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 28s)
  • 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image
  • 01:49 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-text_esams - 7.1.1-2~bpo13+wmf3 ()
  • 00:22 dzahn@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on zuul1005.eqiad.wmnet with reason: host reimage
  • 00:17 dzahn@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on zuul1005.eqiad.wmnet with reason: host reimage
  • 00:01 dzahn@cumin1004: START - Cookbook sre.hosts.reimage for host zuul1005.eqiad.wmnet with OS trixie

2026-09-28

  • 23:44 dzahn@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host zuul1005.eqiad.wmnet with OS trixie
  • 23:35 zabe: run script to fix fr_metadata drifts # T420341
  • 22:22 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
  • 22:21 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
  • 21:47 sbassett: Deployed security mitigation for T435455
  • 21:44 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-text_esams - 7.1.1-2~bpo13+wmf3 ()
  • 21:43 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-upload_esams - 7.1.1-2~bpo13+wmf3 ()
  • 21:42 eevans@deploy1003: helmfile [eqiad] DONE helmfile.d/services/sessionstore: sync
  • 21:42 eevans@deploy1003: helmfile [eqiad] START helmfile.d/services/sessionstore: sync
  • 21:41 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/sessionstore: sync
  • 21:41 eevans@deploy1003: helmfile [staging] START helmfile.d/services/sessionstore: sync
  • 21:24 swfrench-wmf: deleting suspect coredns pods showing upstream resolution health check failures in eqiad - T439201
  • 21:13 cjming@deploy1003: Finished scap sync-world: Backport for fix(ReadingRecommendations): reposition personalize button (T439158), fix(Homepage): apply the body transform only when an overlay is open (T439038) (duration: 12m 47s)
  • 21:12 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on P{cp601[3-6].drmrs.wmnet} and A:cp - 7.1.1-2~bpo13+wmf3 ()
  • 21:08 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
  • 21:08 cjming@deploy1003: volker-e, cjming: Continuing with deployment
  • 21:07 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
  • 21:07 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
  • 21:07 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
  • 21:04 cjming@deploy1003: volker-e, cjming: Backport for fix(ReadingRecommendations): reposition personalize button (T439158), fix(Homepage): apply the body transform only when an overlay is open (T439038) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 21:00 cjming@deploy1003: Started scap sync-world: Backport for fix(ReadingRecommendations): reposition personalize button (T439158), fix(Homepage): apply the body transform only when an overlay is open (T439038)
  • 20:51 kemayo@deploy1003: Finished scap sync-world: Backport for BaseEditCheck#inAllowedNamespace: was true if any extra namespaces set (T400506), EditCheck: only log a session if checks are actually possible (T438916) (duration: 10m 28s)
  • 20:47 herron: uploaded rsyslog 8.2608.0-4~wmf13+1 to trixie-wikimedia T438375
  • 20:47 dzahn@cumin1004: START - Cookbook sre.hosts.reimage for host zuul1005.eqiad.wmnet with OS trixie
  • 20:46 kemayo@deploy1003: kemayo: Continuing with deployment
  • 20:45 kemayo@deploy1003: kemayo: Backport for BaseEditCheck#inAllowedNamespace: was true if any extra namespaces set (T400506), EditCheck: only log a session if checks are actually possible (T438916) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 20:41 kemayo@deploy1003: Started scap sync-world: Backport for BaseEditCheck#inAllowedNamespace: was true if any extra namespaces set (T400506), EditCheck: only log a session if checks are actually possible (T438916)
  • 20:32 cjming@deploy1003: Finished scap sync-world: Backport for Revert "HtmlToContentTransform: Pass DomPageBundle to Parsoid" (duration: 09m 47s)
  • 20:29 dzahn@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host zuul1007.eqiad.wmnet with OS trixie
  • 20:26 cjming@deploy1003: ssastry, cjming: Continuing with deployment
  • 20:26 cjming@deploy1003: ssastry, cjming: Backport for Revert "HtmlToContentTransform: Pass DomPageBundle to Parsoid" synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 20:22 cjming@deploy1003: Started scap sync-world: Backport for Revert "HtmlToContentTransform: Pass DomPageBundle to Parsoid"
  • 20:16 aude@deploy1003: Finished scap sync-world: Backport for Enable ReadingLists on all Wikipedia wikis (T434924), Enable Reading Recommendations experiment featured category setting (T437348) (duration: 13m 05s)
  • 20:13 dzahn@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on zuul1007.eqiad.wmnet with reason: host reimage
  • 20:10 aude@deploy1003: aude, volker-e: Continuing with deployment
  • 20:08 dzahn@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on zuul1007.eqiad.wmnet with reason: host reimage
  • 20:08 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp6013.*
  • 20:07 aude@deploy1003: aude, volker-e: Backport for Enable ReadingLists on all Wikipedia wikis (T434924), Enable Reading Recommendations experiment featured category setting (T437348) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 20:03 aude@deploy1003: Started scap sync-world: Backport for Enable ReadingLists on all Wikipedia wikis (T434924), Enable Reading Recommendations experiment featured category setting (T437348)
  • 19:54 dzahn@cumin1004: START - Cookbook sre.hosts.reimage for host zuul1007.eqiad.wmnet with OS trixie
  • 19:44 dancy@deploy1003: Installation of scap version "4.293.1" completed for 3 hosts
  • 19:42 dancy@deploy1003: Installing scap version "4.293.1" for 3 host(s)
  • 19:31 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P{cp601[3-6].drmrs.wmnet} and A:cp - 7.1.1-2~bpo13+wmf3 ()
  • 19:16 brett@cumin1004: END (FAIL) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=1) rolling upgrade of Varnish on A:cp-text_drmrs - 7.1.1-2~bpo13+wmf3 ()
  • 19:03 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-upload_esams - 7.1.1-2~bpo13+wmf3 ()
  • 19:03 cdobbins@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ncredir5004.eqsin.wmnet with OS trixie
  • 18:47 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-upload_drmrs - 7.1.1-2~bpo13+wmf3 ()
  • 18:46 jhathaway: puppet lvm upgrade broke ml-serve hosts, looking for a hotfix, or I'll rollback...
  • 18:38 dancy@deploy1003: Installation of scap version "4.293.0" completed for 3 hosts
  • 18:36 dancy@deploy1003: Installing scap version "4.293.0" for 3 host(s)
  • 18:24 dzahn@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host zuul1006.eqiad.wmnet with OS trixie
  • 18:19 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir5004.eqsin.wmnet with OS trixie
  • 18:08 dzahn@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on zuul1006.eqiad.wmnet with reason: host reimage
  • 18:04 dzahn@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on zuul1006.eqiad.wmnet with reason: host reimage
  • 17:48 dzahn@cumin1004: START - Cookbook sre.hosts.reimage for host zuul1006.eqiad.wmnet with OS trixie
  • 16:20 zabe@deploy1003: Finished scap sync-world: Backport for wikibase: Set virtual-wikibase-entityusage to x1 for test wikis (T439319) (duration: 13m 50s)
  • 16:15 zabe@deploy1003: zabe: Continuing with deployment
  • 16:10 zabe@deploy1003: zabe: Backport for wikibase: Set virtual-wikibase-entityusage to x1 for test wikis (T439319) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 16:06 zabe@deploy1003: Started scap sync-world: Backport for wikibase: Set virtual-wikibase-entityusage to x1 for test wikis (T439319)
  • 16:05 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-upload_drmrs - 7.1.1-2~bpo13+wmf3 ()
  • 16:04 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-text_drmrs - 7.1.1-2~bpo13+wmf3 ()
  • 16:03 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthbook: apply
  • 16:02 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook: apply
  • 16:01 jdrewniak@deploy1003: Finished scap sync-world: Backport for Bumping portals to master (T128546) (duration: 19m 54s)
  • 15:56 jdrewniak@deploy1003: jdrewniak: Continuing with deployment
  • 15:46 zabe: clone wbc_entity_usage to x1 for testwikidatawiki clients # T439319
  • 15:45 jdrewniak@deploy1003: jdrewniak: Backport for Bumping portals to master (T128546) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 15:41 elukey@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=puppetboard,name=codfw
  • 15:41 elukey@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=puppetdb-api,name=codfw
  • 15:41 jdrewniak@deploy1003: Started scap sync-world: Backport for Bumping portals to master (T128546)
  • 15:40 elukey@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=pki,name=codfw
  • 15:06 filippo@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudcephosd1055.eqiad.wmnet with OS bookworm
  • 14:55 Lucas_WMDE: UTC afternoon backport+config window done
  • 14:54 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for feat(AccountSetup), styles: Increase thumbnail size in ReadingRecommendations (T436332), feat(AccountSetup), styles: Apply correct background and button styles (T436332) (duration: 33m 44s)
  • 14:49 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, volker-e: Continuing with deployment
  • 14:24 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, volker-e: Backport for feat(AccountSetup), styles: Increase thumbnail size in ReadingRecommendations (T436332), feat(AccountSetup), styles: Apply correct background and button styles (T436332) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 14:20 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for feat(AccountSetup), styles: Increase thumbnail size in ReadingRecommendations (T436332), feat(AccountSetup), styles: Apply correct background and button styles (T436332)
  • 14:17 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
  • 14:17 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
  • 14:10 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 14:10 jclark@cumin1004: START - Cookbook sre.hosts.provision for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 14:09 jclark@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 14:09 jclark@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding krb1002 to eqiad - jclark@cumin1004"
  • 14:09 jclark@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding krb1002 to eqiad - jclark@cumin1004"
  • 14:09 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for Revert "Enable Extension:Translate and Extension:TranslationNotifications on co.wikimedia.org" (T386776) (duration: 10m 40s)
  • 14:05 jclark@cumin1004: START - Cookbook sre.dns.netbox
  • 14:03 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde: Continuing with deployment
  • 14:02 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 14:02 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde: Backport for Revert "Enable Extension:Translate and Extension:TranslationNotifications on co.wikimedia.org" (T386776) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 13:58 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for Revert "Enable Extension:Translate and Extension:TranslationNotifications on co.wikimedia.org" (T386776)
  • 13:58 vgutierrez: repool cp7001 - T439361
  • 13:57 jclark@cumin1004: START - Cookbook sre.hosts.provision for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 13:57 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for Enable Extension:Translate and Extension:TranslationNotifications on co.wikimedia.org (T386776) (duration: 19m 38s)
  • 13:54 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 13:54 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, ameisenigel: Rolling back deployment
  • 13:53 jclark@cumin1004: START - Cookbook sre.hosts.provision for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 13:51 tappof: Rebalancing is in progress on kafka-logging in eqiad to evacuate kafka-logging1001 and kafka-logging1002 - T432444
  • 13:47 bking@dns2004: END - running authdns-update
  • 13:44 bking@dns2004: START - running authdns-update
  • 13:42 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 13:41 jclark@cumin1004: START - Cookbook sre.hosts.provision for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 13:41 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, ameisenigel: Backport for Enable Extension:Translate and Extension:TranslationNotifications on co.wikimedia.org (T386776) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 13:37 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for Enable Extension:Translate and Extension:TranslationNotifications on co.wikimedia.org (T386776)
  • 13:36 dpogorzelski@cumin1004: conftool action : set/pooled=true; selector: dnsdisc=k8s-ingress-ml-staging,name=codfw
  • 13:35 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for Enable FlaggedRevs for the Author NS on ruwikisource (T439159) (duration: 12m 02s)
  • 13:33 filippo@cumin1004: START - Cookbook sre.hosts.reimage for host cloudcephosd1055.eqiad.wmnet with OS bookworm
  • 13:33 vgutierrez: depool cp7001 for some purged tests - T439361
  • 13:32 dpogorzelski@cumin1004: conftool action : set/pooled=true; selector: dnsdisc=k8s-ingress-ml-serve,name=codfw
  • 13:30 dpogorzelski@cumin1004: conftool action : set/pooled=true; selector: dnsdisc=inference,name=codfw
  • 13:30 lucaswerkmeister-wmde@deploy1003: ameisenigel, lucaswerkmeister-wmde: Continuing with deployment
  • 13:27 lucaswerkmeister-wmde@deploy1003: ameisenigel, lucaswerkmeister-wmde: Backport for Enable FlaggedRevs for the Author NS on ruwikisource (T439159) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 13:23 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for Enable FlaggedRevs for the Author NS on ruwikisource (T439159)
  • 13:22 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 13:21 jclark@cumin1004: START - Cookbook sre.hosts.provision for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 13:19 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 13:18 jclark@cumin1004: START - Cookbook sre.hosts.provision for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 13:16 sbisson@deploy1003: Finished scap sync-world: Backport for Remove unused config: ArticleGuidanceRedirectEntryPointTitles (T437643) (duration: 12m 58s)
  • 13:11 sbisson@deploy1003: sbisson: Continuing with deployment
  • 13:07 sbisson@deploy1003: sbisson: Backport for Remove unused config: ArticleGuidanceRedirectEntryPointTitles (T437643) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 13:03 sbisson@deploy1003: Started scap sync-world: Backport for Remove unused config: ArticleGuidanceRedirectEntryPointTitles (T437643)
  • 12:33 brouberol@cumin1004: conftool action : set/pooled=yes; selector: name=dse-k8s-worker2008.codfw.wmnet
  • 12:33 brouberol@cumin1004: conftool action : set/weight=10; selector: name=dse-k8s-worker2008.codfw.wmnet
  • 12:33 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2008.codfw.wmnet
  • 12:29 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync
  • 12:29 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync
  • 12:28 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync
  • 12:27 brouberol@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2008.codfw.wmnet
  • 12:21 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker2008.codfw.wmnet with OS bookworm
  • 12:19 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync
  • 12:19 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-text_esams and A:cp - 3.2.23 upgrade (T438828)
  • 12:18 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
  • 12:18 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
  • 12:17 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'.
  • 12:17 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'.
  • 12:15 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply
  • 12:14 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply
  • 12:14 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-liftwing-studio: apply
  • 12:14 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-liftwing-studio: apply
  • 12:04 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker2008.codfw.wmnet with reason: host reimage
  • 11:57 brouberol@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker2008.codfw.wmnet with reason: host reimage
  • 11:54 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync
  • 11:53 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync
  • 11:53 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-web: apply
  • 11:53 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-web: apply
  • 11:53 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-web: apply
  • 11:53 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-web: apply
  • 11:49 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-upload_esams and A:cp - 3.2.23 upgrade (T438828)
  • 11:46 brouberol@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker2008.codfw.wmnet with OS bookworm
  • 11:43 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ganeti-jumbo2003 to dse-k8s-worker2008
  • 11:43 brouberol@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker2008
  • 11:43 brouberol@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker2008
  • 11:43 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker2008 on all recursors
  • 11:43 brouberol@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker2008 on all recursors
  • 11:43 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 11:42 brouberol@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo2003 to dse-k8s-worker2008 - brouberol@cumin1004"
  • 11:42 brouberol@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo2003 to dse-k8s-worker2008 - brouberol@cumin1004"
  • 11:28 zabe@deploy1003: Finished scap sync-world: Backport for reconcileTables: Add sleep option (T438750) (duration: 10m 52s)
  • 11:23 brouberol@cumin1004: START - Cookbook sre.dns.netbox
  • 11:22 brouberol@cumin1004: START - Cookbook sre.hosts.rename from ganeti-jumbo2003 to dse-k8s-worker2008
  • 11:22 zabe@deploy1003: zabe: Continuing with deployment
  • 11:22 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2007.codfw.wmnet
  • 11:21 zabe@deploy1003: zabe: Backport for reconcileTables: Add sleep option (T438750) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 11:20 brouberol@cumin1004: conftool action : set/weight=10; selector: name=dse-k8s-worker2007.codfw.wmnet
  • 11:20 brouberol@cumin1004: conftool action : set/pooled=yes; selector: name=dse-k8s-worker2007.codfw.wmnet
  • 11:17 zabe@deploy1003: Started scap sync-world: Backport for reconcileTables: Add sleep option (T438750)
  • 11:17 brouberol@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2007.codfw.wmnet
  • 11:15 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_esams and A:cp - 3.2.23 upgrade (T438828)
  • 11:15 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_esams and A:cp - 3.2.23 upgrade (T438828)
  • 11:14 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-text_eqiad and A:cp - 3.2.23 upgrade (T438828)
  • 10:49 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-upload_eqiad and A:cp - 3.2.23 upgrade (T438828)
  • 10:43 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/proton: apply
  • 10:43 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.roll-reimage-nodes (exit_code=0) rolling reimage on P{wikikube-worker2041.codfw.wmnet} and (A:wikikube-master-codfw or A:wikikube-worker-codfw)
  • 10:43 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2041.codfw.wmnet
  • 10:43 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2041.codfw.wmnet
  • 10:42 jayme@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2041.codfw.wmnet with OS trixie
  • 10:42 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/proton: apply
  • 10:41 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/proton: apply
  • 10:30 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-web: apply
  • 10:30 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/proton: apply
  • 10:28 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-web: apply
  • 10:28 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-web: apply
  • 10:27 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/proton: apply
  • 10:26 jmm@deploy1003: helmfile [staging] START helmfile.d/services/proton: apply
  • 10:25 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-web: apply
  • 10:25 moritzm: installing chromium security updates
  • 10:23 jayme@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2041.codfw.wmnet with reason: host reimage
  • 10:19 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_eqiad and A:cp - 3.2.23 upgrade (T438828)
  • 10:19 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_eqiad and A:cp - 3.2.23 upgrade (T438828)
  • 10:18 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-text_drmrs and A:cp - 3.2.23 upgrade (T438828)
  • 10:17 jayme@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2041.codfw.wmnet with reason: host reimage
  • 10:12 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker2007.codfw.wmnet with OS bookworm
  • 10:00 jayme@cumin1004: START - Cookbook sre.hosts.reimage for host wikikube-worker2041.codfw.wmnet with OS trixie
  • 10:00 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2041.codfw.wmnet
  • 09:54 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2041.codfw.wmnet
  • 09:54 jayme@cumin1004: START - Cookbook sre.k8s.roll-reimage-nodes rolling reimage on P{wikikube-worker2041.codfw.wmnet} and (A:wikikube-master-codfw or A:wikikube-worker-codfw)
  • 09:53 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker2007.codfw.wmnet with reason: host reimage
  • 09:50 brouberol@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker2007.codfw.wmnet with reason: host reimage
  • 09:46 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply
  • 09:46 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-upload_drmrs and A:cp - 3.2.23 upgrade (T438828)
  • 09:45 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply
  • 09:39 brouberol@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker2007.codfw.wmnet with OS bookworm
  • 09:36 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ganeti-jumbo2002 to dse-k8s-worker2007
  • 09:36 brouberol@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker2007
  • 09:35 brouberol@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker2007
  • 09:35 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker2007 on all recursors
  • 09:35 brouberol@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker2007 on all recursors
  • 09:35 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 09:35 brouberol@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo2002 to dse-k8s-worker2007 - brouberol@cumin1004"
  • 09:35 brouberol@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo2002 to dse-k8s-worker2007 - brouberol@cumin1004"
  • 09:31 brouberol@cumin1004: START - Cookbook sre.dns.netbox
  • 09:31 brouberol@cumin1004: START - Cookbook sre.hosts.rename from ganeti-jumbo2002 to dse-k8s-worker2007
  • 09:29 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync
  • 09:27 brouberol@cumin1004: conftool action : set/pooled=yes; selector: name=dse-k8s-worker2006.codfw.wmnet
  • 09:27 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-eqiad@eqiad
  • 09:27 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs
  • 09:27 brouberol@cumin1004: conftool action : set/weight=10; selector: name=dse-k8s-worker2006.codfw.wmnet
  • 09:26 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs
  • 09:26 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2006.codfw.wmnet
  • 09:22 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad
  • 09:20 brouberol@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2006.codfw.wmnet
  • 09:19 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync
  • 09:10 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-codfw@codfw
  • 09:10 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs
  • 09:10 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_drmrs and A:cp - 3.2.23 upgrade (T438828)
  • 09:10 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_drmrs and A:cp - 3.2.23 upgrade (T438828)
  • 09:09 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs
  • 09:05 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-codfw@codfw
  • 09:04 dkertesz: running ncmonitor manually on ncmonitor1001 since last execution failed due to a 504 from MarkMonitor
  • 09:03 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync
  • 09:00 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
  • 09:00 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
  • 08:52 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync
  • 08:52 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-liftwing-studio: sync
  • 08:52 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-liftwing-studio: sync
  • 08:51 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/ferretdb-liftwing-studio: sync
  • 08:51 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/ferretdb-liftwing-studio: sync
  • 08:50 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker2006.codfw.wmnet with OS bookworm
  • 08:44 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync
  • 08:36 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync
  • 08:33 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker2006.codfw.wmnet with reason: host reimage
  • 08:29 brouberol@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker2006.codfw.wmnet with reason: host reimage
  • 08:26 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-text_codfw and not P{cp2059.*} and A:cp - 3.2.23 upgrade (T438828)
  • 08:18 brouberol@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker2006.codfw.wmnet with OS bookworm
  • 08:18 brouberol@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host dse-k8s-worker2006.codfw.wmnet with OS bookworm
  • 08:16 moritzm: installing Exim security updates
  • 08:04 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-upload_codfw and A:cp - 3.2.23 upgrade (T438828)
  • 07:59 matthiasmullie: UTC morning backport window complete
  • 07:59 mlitn@deploy1003: Finished scap sync-world: Backport for Growth: Set minimum registration date for A/B test (T432126), EarlyOnboarding: record account setup motivation in `action_context` (T437339) (duration: 14m 12s)
  • 07:58 brouberol@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker2006.codfw.wmnet with OS bookworm
  • 07:53 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ganeti-jumbo2001 to dse-k8s-worker2006
  • 07:52 brouberol@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker2006
  • 07:52 brouberol@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker2006
  • 07:52 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker2006 on all recursors
  • 07:52 brouberol@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker2006 on all recursors
  • 07:52 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 07:52 brouberol@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo2001 to dse-k8s-worker2006 - brouberol@cumin1004"
  • 07:52 brouberol@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo2001 to dse-k8s-worker2006 - brouberol@cumin1004"
  • 07:51 mlitn@deploy1003: migr, mlitn: Continuing with deployment
  • 07:49 mlitn@deploy1003: migr, mlitn: Backport for Growth: Set minimum registration date for A/B test (T432126), EarlyOnboarding: record account setup motivation in `action_context` (T437339) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 07:48 brouberol@cumin1004: START - Cookbook sre.dns.netbox
  • 07:47 brouberol@cumin1004: START - Cookbook sre.hosts.rename from ganeti-jumbo2001 to dse-k8s-worker2006
  • 07:44 mlitn@deploy1003: Started scap sync-world: Backport for Growth: Set minimum registration date for A/B test (T432126), EarlyOnboarding: record account setup motivation in `action_context` (T437339)
  • 07:40 mlitn@deploy1003: Finished scap sync-world: Backport for [MultimediaViewer] Testwiki use same carousel config as other wikis, Fix jump to image break in carousel retest (T439252), Remove reliance on ParserOutput (T439182), Remove reliance on file repo (T439182) (duration: 32m 06s)
  • 07:38 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_codfw and A:cp - 3.2.23 upgrade (T438828)
  • 07:38 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_codfw and not P{cp2059.*} and A:cp - 3.2.23 upgrade (T438828)
  • 07:27 mlitn@deploy1003: mlitn: Continuing with deployment
  • 07:26 mlitn@deploy1003: mlitn: Backport for [MultimediaViewer] Testwiki use same carousel config as other wikis, Fix jump to image break in carousel retest (T439252), Remove reliance on ParserOutput (T439182), Remove reliance on file repo (T439182) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verifi
  • 07:08 mlitn@deploy1003: Started scap sync-world: Backport for [MultimediaViewer] Testwiki use same carousel config as other wikis, Fix jump to image break in carousel retest (T439252), Remove reliance on ParserOutput (T439182), Remove reliance on file repo (T439182)
  • 06:28 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply
  • 06:27 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply
  • 06:10 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/ferretdb-growthbook-next: apply
  • 06:10 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/ferretdb-growthbook-next: apply
  • 06:09 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/ferretdb-growthbook: apply
  • 06:09 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/ferretdb-growthbook: apply
  • 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 48s)
  • 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image

2026-09-27

  • 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 32s)
  • 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image

2026-09-26

  • 21:26 krinkle@deploy1003: Finished deploy [performance/arc-lamp@68349ee]: https://gerrit.wikimedia.org/r/c/performance/arc-lamp/+/1345296 (duration: 00m 09s)
  • 21:26 krinkle@deploy1003: Started deploy [performance/arc-lamp@68349ee]: https://gerrit.wikimedia.org/r/c/performance/arc-lamp/+/1345296
  • 16:37 ssastry@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
  • 16:37 ssastry@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
  • 16:37 ssastry@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
  • 16:37 ssastry@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
  • 16:30 ssastry@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
  • 16:30 ssastry@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
  • 16:30 ssastry@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
  • 16:29 ssastry@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
  • 08:07 oblivian@deploy1003: Finished scap sync-world: Backport for Revert "Disable Score exec" (duration: 10m 53s)
  • 08:02 oblivian@deploy1003: oblivian: Continuing with deployment
  • 08:00 oblivian@deploy1003: oblivian: Backport for Revert "Disable Score exec" synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 07:56 oblivian@deploy1003: Started scap sync-world: Backport for Revert "Disable Score exec"
  • 07:52 oblivian@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
  • 07:50 oblivian@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
  • 07:46 oblivian@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
  • 07:44 oblivian@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply
  • 07:42 oblivian@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
  • 07:42 oblivian@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply
  • 06:30 oblivian@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox: apply
  • 06:30 oblivian@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox: apply
  • 06:29 oblivian@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: apply
  • 06:29 oblivian@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: apply
  • 06:28 oblivian@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox: apply
  • 06:27 oblivian@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox: apply
  • 03:37 ladsgroup@deploy1003: Finished scap sync-world: Backport for Disable Score exec (T439297 T438443) (duration: 11m 01s)
  • 03:31 ladsgroup@deploy1003: ladsgroup: Continuing with deployment
  • 03:30 ladsgroup@deploy1003: ladsgroup: Backport for Disable Score exec (T439297 T438443) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 03:26 ladsgroup@deploy1003: Started scap sync-world: Backport for Disable Score exec (T439297 T438443)
  • 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 13s)
  • 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image

2026-09-25

  • 23:15 jclark@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudcephosd1055.eqiad.wmnet with OS bookworm
  • 22:51 ssastry@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
  • 22:51 ssastry@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
  • 22:51 ssastry@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
  • 22:51 ssastry@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
  • 22:47 ssastry@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
  • 22:46 ssastry@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
  • 22:46 ssastry@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
  • 22:46 ssastry@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
  • 22:39 jclark@cumin1004: START - Cookbook sre.hosts.reimage for host cloudcephosd1055.eqiad.wmnet with OS bookworm
  • 18:27 krinkle@deploy1003: Finished deploy [statsv/statsv@df3ebff]: T439183: Accept dot, plus, hyphen in label values (duration: 00m 11s)
  • 18:27 krinkle@deploy1003: Started deploy [statsv/statsv@df3ebff]: T439183: Accept dot, plus, hyphen in label values
  • 17:59 cdanis@dns1004: END - running authdns-update
  • 17:57 cdanis@dns1004: START - running authdns-update
  • 15:07 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir2001.*
  • 15:03 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
  • 15:03 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
  • 15:02 dcausse@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search: apply
  • 15:01 dcausse@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search: apply
  • 15:01 dcausse@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search: apply
  • 15:01 dcausse@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-semantic-search: apply
  • 14:57 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir2001.codfw.wmnet with OS trixie
  • 14:56 brouberol@cumin1004: conftool action : set/weight=10; selector: name=dse-k8s-worker1017.eqiad.wmnet
  • 14:56 brouberol@cumin1004: conftool action : set/pooled=yes; selector: name=dse-k8s-worker1017.eqiad.wmnet
  • 14:51 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1040.eqiad.wmnet
  • 14:51 brouberol@cumin1004: conftool action : set/pooled=yes; selector: name=dse-k8s-worker1040.eqiad.wmnet
  • 14:51 brouberol@cumin1004: conftool action : set/weight=10; selector: name=dse-k8s-worker1040.eqiad.wmnet
  • 14:49 brouberol@cumin1004: conftool action : set/weight=10; selector: name=dse-k8s-worker1041.eqiad.wmnet
  • 14:49 brouberol@cumin1004: conftool action : set/pooled=yes; selector: name=dse-k8s-worker1041.eqiad.wmnet
  • 14:49 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1041.eqiad.wmnet
  • 14:46 brouberol@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1040.eqiad.wmnet
  • 14:44 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
  • 14:44 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
  • 14:43 brouberol@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1041.eqiad.wmnet
  • 14:41 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
  • 14:41 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
  • 14:38 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir2001.codfw.wmnet with reason: host reimage
  • 14:33 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir2001.codfw.wmnet with reason: host reimage
  • 14:32 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1040.eqiad.wmnet with OS bookworm
  • 14:30 dkertesz: moved haproxy stat file from /var/lib/haproxy/stats-file to /run/haproxy/ in cp7001,cp7011 - T343000
  • 14:29 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1041.eqiad.wmnet with OS bookworm
  • 14:23 vgutierrez@puppetserver1001: conftool action : set/pooled=yes; selector: dc=codfw,name=cp2059.*
  • 14:18 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wmde: apply
  • 14:17 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wmde: apply
  • 14:17 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wikidata: apply
  • 14:17 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wikidata: apply
  • 14:17 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-sre: apply
  • 14:17 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-sre: apply
  • 14:15 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-search: apply
  • 14:15 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-search: apply
  • 14:14 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-research: apply
  • 14:14 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-research: apply
  • 14:14 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir2001.codfw.wmnet with OS trixie
  • 14:13 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-platform-eng: apply
  • 14:13 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-platform-eng: apply
  • 14:13 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
  • 14:13 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply
  • 14:06 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1040.eqiad.wmnet with reason: host reimage
  • 14:06 brouberol@cumin1004: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on dse-k8s-worker1041.eqiad.wmnet with reason: host reimage
  • 14:05 brouberol@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1041.eqiad.wmnet with reason: host reimage
  • 14:02 brouberol@cumin1004: conftool action : set/weight=10; selector: name=dse-k8s-worker1039.eqiad.wmnet
  • 14:01 brouberol@cumin1004: conftool action : set/pooled=yes; selector: name=dse-k8s-worker1039.eqiad.wmnet
  • 14:00 atsuko@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=eventgate-main,name=codfw
  • 14:00 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
  • 14:00 atsuko@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=eventgate-logging-external,name=codfw
  • 14:00 atsuko@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=eventgate-analytics-external,name=codfw
  • 14:00 atsuko@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=eventgate-analytics,name=codfw
  • 14:00 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
  • 13:59 brouberol@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1040.eqiad.wmnet with reason: host reimage
  • 13:58 dcausse@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply
  • 13:58 dcausse@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply
  • 13:55 dcausse@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply
  • 13:55 dcausse@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply
  • 13:54 brouberol@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1041.eqiad.wmnet with OS bookworm
  • 13:53 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ganeti-jumbo1003 to dse-k8s-worker1041
  • 13:53 brouberol@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1041
  • 13:52 brouberol@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1041
  • 13:52 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1041 on all recursors
  • 13:52 brouberol@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker1041 on all recursors
  • 13:52 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 13:52 brouberol@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo1003 to dse-k8s-worker1041 - brouberol@cumin1004"
  • 13:52 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
  • 13:52 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
  • 13:51 brouberol@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo1003 to dse-k8s-worker1041 - brouberol@cumin1004"
  • 13:51 zabe: clone wbc_entity_usage from local cluster to x1 for all wikidata client wikis # T438750
  • 13:50 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1039.eqiad.wmnet
  • 13:49 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply
  • 13:48 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply
  • 13:47 brouberol@cumin1004: START - Cookbook sre.dns.netbox
  • 13:47 brouberol@cumin1004: START - Cookbook sre.hosts.rename from ganeti-jumbo1003 to dse-k8s-worker1041
  • 13:46 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on P{lvs1019.*} and A:lvs
  • 13:46 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on P{lvs1019.*} and A:lvs
  • 13:45 brouberol@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1039.eqiad.wmnet
  • 13:45 brouberol@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1040.eqiad.wmnet with OS bookworm
  • 13:44 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on P{lvs1020.*} and A:lvs
  • 13:44 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on P{lvs1020.*} and A:lvs
  • 13:42 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-fr-tech: apply
  • 13:42 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-fr-tech: apply
  • 13:40 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ganeti-jumbo1002 to dse-k8s-worker1040
  • 13:39 brouberol@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1040
  • 13:39 brouberol@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1040
  • 13:39 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1040 on all recursors
  • 13:39 brouberol@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker1040 on all recursors
  • 13:39 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 13:39 brouberol@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo1002 to dse-k8s-worker1040 - brouberol@cumin1004"
  • 13:38 brouberol@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo1002 to dse-k8s-worker1040 - brouberol@cumin1004"
  • 13:34 brouberol@cumin1004: START - Cookbook sre.dns.netbox
  • 13:34 brouberol@cumin1004: START - Cookbook sre.hosts.rename from ganeti-jumbo1002 to dse-k8s-worker1040
  • 13:29 mvernon@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool sessionstore in codfw: return to active/active
  • 13:24 Emperor: repool sessionstore in codfw
  • 13:24 mvernon@cumin1004: START - Cookbook sre.discovery.service-route pool sessionstore in codfw: return to active/active
  • 13:24 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
  • 13:24 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
  • 13:22 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-experiment-platform: apply
  • 13:22 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-experiment-platform: apply
  • 13:20 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
  • 13:20 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
  • 13:15 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1039.eqiad.wmnet with OS bookworm
  • 13:03 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1152.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 12:59 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp2049.codfw.wmnet
  • 12:58 jclark@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1152.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 12:55 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1039.eqiad.wmnet with reason: host reimage
  • 12:52 brouberol@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1039.eqiad.wmnet with reason: host reimage
  • 12:47 mvernon@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) check sessionstore: maintenance
  • 12:47 mvernon@cumin1004: START - Cookbook sre.discovery.service-route check sessionstore: maintenance
  • 12:45 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'.
  • 12:44 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'.
  • 12:44 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
  • 12:43 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
  • 12:42 brouberol@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1039.eqiad.wmnet with OS bookworm
  • 12:40 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ganeti-jumbo1001 to dse-k8s-worker1039
  • 12:40 brouberol@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1039
  • 12:39 brouberol@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1039
  • 12:39 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1039 on all recursors
  • 12:39 brouberol@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker1039 on all recursors
  • 12:39 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 12:39 brouberol@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo1001 to dse-k8s-worker1039 - brouberol@cumin1004"
  • 12:38 brouberol@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo1001 to dse-k8s-worker1039 - brouberol@cumin1004"
  • 12:34 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cumin1003.eqiad.wmnet
  • 12:34 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 12:34 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin1003.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003"
  • 12:34 brouberol@cumin1004: START - Cookbook sre.dns.netbox
  • 12:33 brouberol@cumin1004: START - Cookbook sre.hosts.rename from ganeti-jumbo1001 to dse-k8s-worker1039
  • 12:26 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin1003.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003"
  • 12:21 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply
  • 12:21 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply
  • 12:20 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-product: apply
  • 12:20 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-product: apply
  • 12:18 jmm@cumin2003: START - Cookbook sre.dns.netbox
  • 12:13 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cumin1003.eqiad.wmnet
  • 11:41 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-ctrl1002.eqiad.wmnet
  • 11:40 urbanecm@deploy1003: mwscript-k8s job started: foreachwikiindblist growthexperiments GrowthExperiments:revalidateLinkRecommendations.php --olderThan=1790175600 --verbose # T438366
  • 11:36 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-ctrl1002.eqiad.wmnet
  • 11:20 kevinbazira@deploy1003: helmfile [ml-serve-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' .
  • 11:19 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'tts-section-generator' for release 'main' .
  • 11:17 kevinbazira@deploy1003: helmfile [ml-staging-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' .
  • 10:58 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-ctrl1001.eqiad.wmnet
  • 10:54 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
  • 10:54 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
  • 10:53 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-ctrl1001.eqiad.wmnet
  • 10:52 jelto@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4 days, 0:00:00 on wikikube-worker1152.eqiad.wmnet with reason: hardware/networking issues
  • 09:49 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from codfw to eqiad for section test-s4
  • 09:49 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.finalize for the switch from codfw to eqiad for section test-s4
  • 09:49 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from codfw to eqiad for section test-s4
  • 09:48 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from codfw to eqiad for section test-s4
  • 09:43 tappof: reset modified_attributes for hosts and services that fully match the Puppet configuration in Icinga - T439105
  • 09:36 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from eqiad to codfw for section test-s4
  • 09:36 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.finalize for the switch from eqiad to codfw for section test-s4
  • 09:36 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section test-s4
  • 09:35 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section test-s4
  • 09:28 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts build2001.codfw.wmnet
  • 09:28 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 09:28 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: build2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003"
  • 09:11 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-worker1207.eqiad.wmnet
  • 09:01 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: build2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003"
  • 08:57 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host an-worker1207.eqiad.wmnet
  • 08:57 jmm@cumin2003: START - Cookbook sre.dns.netbox
  • 08:52 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts build2001.codfw.wmnet
  • 08:24 elukey@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'sync'.
  • 08:23 elukey@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'sync'.
  • 08:23 elukey@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'sync'.
  • 08:22 elukey@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'sync'.
  • 08:20 vgutierrez@puppetserver1001: conftool action : set/weight=1; selector: dc=codfw,name=cp2059.*
  • 08:15 vgutierrez@puppetserver1001: conftool action : set/pooled=no; selector: dc=codfw,name=cp2059.*
  • 05:58 dcausse@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply
  • 05:58 dcausse@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply
  • 05:21 ryankemper: [Cirrus] Stumble across orphaned index `sawikisource_content_1784136042`, deleted. The real index is `sawikisource_content_1784136826` which I've obviously left untouched
  • 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 38s)
  • 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image
  • 01:41 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P{dse-k8s-worker1*.eqiad.wmnet} and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad)
  • 01:41 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1028.eqiad.wmnet
  • 01:41 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1028.eqiad.wmnet
  • 01:30 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1028.eqiad.wmnet
  • 01:00 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1028.eqiad.wmnet
  • 01:00 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1027.eqiad.wmnet
  • 01:00 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1027.eqiad.wmnet
  • 00:53 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1027.eqiad.wmnet
  • 00:53 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1027.eqiad.wmnet
  • 00:53 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1026.eqiad.wmnet
  • 00:53 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1026.eqiad.wmnet
  • 00:44 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1026.eqiad.wmnet
  • 00:14 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1026.eqiad.wmnet
  • 00:14 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1025.eqiad.wmnet
  • 00:14 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1025.eqiad.wmnet
  • 00:07 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1025.eqiad.wmnet
  • 00:07 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1025.eqiad.wmnet
  • 00:06 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1024.eqiad.wmnet
  • 00:06 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1024.eqiad.wmnet

2026-09-24

  • 23:58 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1024.eqiad.wmnet
  • 23:57 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1024.eqiad.wmnet
  • 23:57 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1023.eqiad.wmnet
  • 23:57 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1023.eqiad.wmnet
  • 23:50 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1023.eqiad.wmnet
  • 23:32 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on P{cp404[1-6].ulsfo.wmnet} and A:cp - 7.1.1-2~bpo13+wmf3 ()
  • 23:20 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1023.eqiad.wmnet
  • 23:20 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1022.eqiad.wmnet
  • 23:20 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1022.eqiad.wmnet
  • 23:11 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1022.eqiad.wmnet
  • 22:41 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1022.eqiad.wmnet
  • 22:41 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1021.eqiad.wmnet
  • 22:41 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1021.eqiad.wmnet
  • 22:28 ryankemper: [WDQS] Expanding match in https://requestctl.wikimedia.org/pattern/ua/rocks to test a likely block candidate
  • {{safesubst:SAL entry|1=22:27 egardner@deploy1003: Finished scap sync-world: Backport for ReaderExperiments: Set the preferred-sources debug flag on testwiki (T436692), ReaderExperiments: Drop the stale ShareHighlight config var (T424764), Enable ReadingList CTA on Minerva for our test wikis (inc beta cluster) (T438779), [[gerrit:1343560|Revert "Enable Reading Recommendations experiment on t}}
  • 22:22 egardner@deploy1003: volker-e, egardner, jdlrobson: Continuing with deployment
  • 22:21 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1021.eqiad.wmnet
  • 22:19 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1021.eqiad.wmnet
  • 22:19 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1020.eqiad.wmnet
  • 22:19 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1020.eqiad.wmnet
  • {{safesubst:SAL entry|1=22:14 egardner@deploy1003: volker-e, egardner, jdlrobson: Backport for ReaderExperiments: Set the preferred-sources debug flag on testwiki (T436692), ReaderExperiments: Drop the stale ShareHighlight config var (T424764), Enable ReadingList CTA on Minerva for our test wikis (inc beta cluster) (T438779), [[gerrit:1343560|Revert "Enable Reading Recommendations experiment}}
  • {{safesubst:SAL entry|1=22:10 egardner@deploy1003: Started scap sync-world: Backport for ReaderExperiments: Set the preferred-sources debug flag on testwiki (T436692), ReaderExperiments: Drop the stale ShareHighlight config var (T424764), Enable ReadingList CTA on Minerva for our test wikis (inc beta cluster) (T438779), [[gerrit:1343560|Revert "Enable Reading Recommendations experiment on te}}
  • 22:04 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-text_magru and not P{cp7001.magru.wmnet} and A:cp - 7.1.1-2~bpo13+wmf3 ()
  • 22:02 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1020.eqiad.wmnet
  • 22:00 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1020.eqiad.wmnet
  • 22:00 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1019.eqiad.wmnet
  • 22:00 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1019.eqiad.wmnet
  • 21:58 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp4052.*
  • 21:57 jhuneidi@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.47.0-wmf.21 refs T438217
  • 21:53 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1019.eqiad.wmnet
  • 21:53 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1019.eqiad.wmnet
  • 21:53 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1018.eqiad.wmnet
  • 21:53 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1018.eqiad.wmnet
  • 21:48 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on P{cp4052.ulsfo.wmnet} and A:cp - 7.1.1-2~bpo13+wmf3 ()
  • 21:46 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1018.eqiad.wmnet
  • 21:46 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1018.eqiad.wmnet
  • 21:46 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1016.eqiad.wmnet
  • 21:46 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1016.eqiad.wmnet
  • 21:45 catrope@deploy1003: Finished scap sync-world: Backport for Catch newline character in UserMailer to prevent it from allowing bad actors to create an additional header (T434545) (duration: 17m 05s)
  • 21:42 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P{cp4052.ulsfo.wmnet} and A:cp - 7.1.1-2~bpo13+wmf3 ()
  • 21:40 catrope@deploy1003: catrope: Continuing with deployment
  • 21:35 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1016.eqiad.wmnet
  • 21:35 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1016.eqiad.wmnet
  • 21:34 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1015.eqiad.wmnet
  • 21:34 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1015.eqiad.wmnet
  • 21:34 brett@cumin1004: END (FAIL) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=1) rolling upgrade of Varnish on P{cp405[1-2].ulsfo.wmnet} and A:cp - 7.1.1-2~bpo13+wmf3 ()
  • 21:33 catrope@deploy1003: catrope: Backport for Catch newline character in UserMailer to prevent it from allowing bad actors to create an additional header (T434545) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 21:28 catrope@deploy1003: Started scap sync-world: Backport for Catch newline character in UserMailer to prevent it from allowing bad actors to create an additional header (T434545)
  • 21:28 catrope@deploy1003: Finished scap sync-world: Backport for ext.wikimediaEvents.testKitchen: Add withContext helper (T438898), ReaderExperiments: add dewiki and svwiki (T438072), Image Browsing carousel: taps outside the preview dialog should close it (T439006), Cap the dialog viewport (T439007) (duration: 19m 27s)
  • 21:26 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1015.eqiad.wmnet
  • 21:22 catrope@deploy1003: cjming, mfossati, catrope, mlitn: Continuing with deployment
  • 21:12 catrope@deploy1003: cjming, mfossati, catrope, mlitn: Backport for ext.wikimediaEvents.testKitchen: Add withContext helper (T438898), ReaderExperiments: add dewiki and svwiki (T438072), Image Browsing carousel: taps outside the preview dialog should close it (T439006), Cap the dialog viewport (T439007) synced to the testservers (see https://wi
  • 21:08 catrope@deploy1003: Started scap sync-world: Backport for ext.wikimediaEvents.testKitchen: Add withContext helper (T438898), ReaderExperiments: add dewiki and svwiki (T438072), Image Browsing carousel: taps outside the preview dialog should close it (T439006), Cap the dialog viewport (T439007)
  • 21:04 catrope@deploy1003: Finished scap sync-world: Backport for Revert "cirrus: Send more_like traffic to eqiad", prv: Enable parsoid rendering for 5 wikis (T438998) (duration: 10m 45s)
  • 21:03 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp4051.*
  • 21:02 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp4041.*
  • 20:58 catrope@deploy1003: catrope, ebernhardson, jgiannelos: Continuing with deployment
  • 20:57 catrope@deploy1003: catrope, ebernhardson, jgiannelos: Backport for Revert "cirrus: Send more_like traffic to eqiad", prv: Enable parsoid rendering for 5 wikis (T438998) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 20:57 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P{cp405[1-2].ulsfo.wmnet} and A:cp - 7.1.1-2~bpo13+wmf3 ()
  • 20:56 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1015.eqiad.wmnet
  • 20:56 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1014.eqiad.wmnet
  • 20:56 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1014.eqiad.wmnet
  • 20:55 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
  • 20:55 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
  • 20:53 catrope@deploy1003: Started scap sync-world: Backport for Revert "cirrus: Send more_like traffic to eqiad", prv: Enable parsoid rendering for 5 wikis (T438998)
  • 20:50 brett@cumin1004: END (FAIL) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=1) rolling upgrade of Varnish on P{cp405[1-2].ulsfo.wmnet} and A:cp - 7.1.1-2~bpo13+wmf3 ()
  • 20:49 catrope@deploy1003: Finished scap sync-world: Backport for HookHandler: Guard against recovery code expiry being null (T438593) (duration: 10m 19s)
  • 20:49 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply
  • 20:48 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply
  • 20:48 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply
  • 20:47 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply
  • 20:44 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P{cp405[1-2].ulsfo.wmnet} and A:cp - 7.1.1-2~bpo13+wmf3 ()
  • 20:44 catrope@deploy1003: catrope: Continuing with deployment
  • 20:43 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P{cp404[1-6].ulsfo.wmnet} and A:cp - 7.1.1-2~bpo13+wmf3 ()
  • 20:43 catrope@deploy1003: catrope: Backport for HookHandler: Guard against recovery code expiry being null (T438593) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 20:39 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1014.eqiad.wmnet
  • 20:39 catrope@deploy1003: Started scap sync-world: Backport for HookHandler: Guard against recovery code expiry being null (T438593)
  • 20:34 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
  • 20:34 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
  • 20:25 brett@cumin1004: END (FAIL) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=1) rolling upgrade of Varnish on A:cp-text_ulsfo - 7.1.1-2~bpo13+wmf3 ()
  • 20:25 brett@cumin1004: END (FAIL) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=1) rolling upgrade of Varnish on A:cp-upload_ulsfo - 7.1.1-2~bpo13+wmf3 ()
  • 20:19 kemayo@deploy1003: Finished scap sync-world: Backport for EditCheck: add some statsv tracking of check/suggestion actions (T438916) (duration: 11m 23s)
  • 20:14 kemayo@deploy1003: kemayo: Continuing with deployment
  • 20:12 kemayo@deploy1003: kemayo: Backport for EditCheck: add some statsv tracking of check/suggestion actions (T438916) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 20:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1014.eqiad.wmnet
  • 20:09 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1013.eqiad.wmnet
  • 20:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1013.eqiad.wmnet
  • 20:08 kemayo@deploy1003: Started scap sync-world: Backport for EditCheck: add some statsv tracking of check/suggestion actions (T438916)
  • 20:01 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1013.eqiad.wmnet
  • 19:57 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen: apply
  • 19:56 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen: apply
  • 19:56 cdobbins@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ncredir5004.eqsin.wmnet with OS trixie
  • 19:50 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-upload_magru and not P{cp7011.magru.wmnet} and A:cp - 7.1.1-2~bpo13+wmf3 ()
  • 19:46 vriley@cumin1004: START - Cookbook sre.hosts.reimage for host zuul1005.eqiad.wmnet with OS trixie
  • 19:36 ryankemper: [Cirrus] All cirrus pools are serving again. Actively monitoring while the system returns to equilibrium, but all initial indications are that things are as they should be
  • 19:34 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
  • 19:34 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
  • 19:33 ryankemper@cumin2003: END (FAIL) - Cookbook sre.discovery.service-route (exit_code=99) pool search-omega in codfw: maintenance
  • 19:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1013.eqiad.wmnet
  • 19:31 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1012.eqiad.wmnet
  • 19:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1012.eqiad.wmnet
  • 19:29 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
  • 19:29 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
  • 19:28 ryankemper@cumin2003: START - Cookbook sre.discovery.service-route pool search-omega in codfw: maintenance
  • 19:27 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=k8s-ingress-aux-ro
  • 19:26 ryankemper: [Cirrus] nevermind, that's just the cookbook assuming the DNS record should exist, which it doesn't because chi/psi/omega all share `search.svc.$DC.wmnet`
  • 19:25 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1012.eqiad.wmnet
  • 19:24 ryankemper: [Cirrus] `dns.resolver.NoAnswer: The DNS response does not contain an answer to the question: search-psi.svc.eqiad.wmnet` checking briefly if this is real failure or just some TTL wonkiness
  • 19:23 ryankemper@cumin2003: END (FAIL) - Cookbook sre.discovery.service-route (exit_code=99) pool search-psi in codfw: maintenance
  • 19:20 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
  • 19:20 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
  • 19:18 dzahn@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host zuul1005.eqiad.wmnet with OS trixie
  • 19:18 ryankemper@cumin2003: START - Cookbook sre.discovery.service-route pool search-psi in codfw: maintenance
  • 19:17 ryankemper: [Cirrus] codfw chi (big cluster) repooled; metrics are already improving, I see poolcounter rejections dropping significantly
  • 19:17 ryankemper@cumin2003: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool search in codfw: maintenance
  • 19:17 cdanis@cumin1004: conftool action : set/ttl=300; selector: name=codfw
  • 19:13 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir5004.eqsin.wmnet with OS trixie
  • 19:12 ryankemper@cumin2003: START - Cookbook sre.discovery.service-route pool search in codfw: maintenance
  • 19:11 ryankemper: [Cirrus] Repooling codfw, chi first followed by the small clusters
  • 19:11 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=(kartotherian|tegola-vector-tiles)
  • 19:07 cdobbins@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ncredir5004.eqsin.wmnet with OS trixie
  • 19:02 jhuneidi@deploy1003: Finished scap sync-world: Backport for REST: restore PageContentHelper::checkAccess (fix live breakage) (duration: 10m 15s)
  • 18:57 jhuneidi@deploy1003: daniel, jhuneidi: Continuing with deployment
  • 18:56 jhuneidi@deploy1003: daniel, jhuneidi: Backport for REST: restore PageContentHelper::checkAccess (fix live breakage) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 18:55 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1012.eqiad.wmnet
  • 18:55 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1011.eqiad.wmnet
  • 18:55 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1011.eqiad.wmnet
  • 18:52 jhuneidi@deploy1003: Started scap sync-world: Backport for REST: restore PageContentHelper::checkAccess (fix live breakage)
  • 18:49 ryankemper@cumin2003: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool wdqs-internal-scholarly in codfw: maintenance
  • 18:49 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1011.eqiad.wmnet
  • 18:48 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1011.eqiad.wmnet
  • 18:48 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1010.eqiad.wmnet
  • 18:48 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1010.eqiad.wmnet
  • 18:44 ryankemper@cumin2003: START - Cookbook sre.discovery.service-route pool wdqs-internal-scholarly in codfw: maintenance
  • 18:44 ryankemper@cumin2003: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool wdqs-internal-main in codfw: maintenance
  • 18:42 herron@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=thanos-swift,name=codfw
  • 18:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply
  • 18:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply
  • 18:40 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1010.eqiad.wmnet
  • 18:39 herron@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=thanos-query,name=codfw
  • 18:39 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply
  • 18:39 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1010.eqiad.wmnet
  • 18:39 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1009.eqiad.wmnet
  • 18:39 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1009.eqiad.wmnet
  • 18:39 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply
  • 18:39 ryankemper@cumin2003: START - Cookbook sre.discovery.service-route pool wdqs-internal-main in codfw: maintenance
  • 18:38 ryankemper@cumin2003: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool wcqs in codfw: maintenance
  • 18:37 herron@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=thanos-web.*,name=codfw
  • 18:36 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply
  • 18:34 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply
  • 18:34 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply
  • 18:33 ryankemper@cumin2003: START - Cookbook sre.discovery.service-route pool wcqs in codfw: maintenance
  • 18:33 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply
  • 18:33 ryankemper@cumin2003: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool wdqs-scholarly in codfw: maintenance
  • 18:31 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1009.eqiad.wmnet
  • 18:30 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
  • 18:29 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
  • 18:28 ryankemper@cumin2003: START - Cookbook sre.discovery.service-route pool wdqs-scholarly in codfw: maintenance
  • 18:25 ryankemper@cumin2003: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool wdqs-main in codfw: maintenance
  • 18:25 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir5004.eqsin.wmnet with OS trixie
  • 18:20 ryankemper@cumin2003: START - Cookbook sre.discovery.service-route pool wdqs-main in codfw: maintenance
  • 18:19 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply
  • 18:19 jhuneidi@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.21 refs T438217
  • 18:19 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply
  • 18:18 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply
  • 18:18 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply
  • 18:17 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply
  • 18:16 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply
  • 18:15 ryankemper: [WDQS] Preparing to repool codfw WDQS shortly; it's been operating single DC so this second DC should restore proper service availability
  • 18:13 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply
  • 18:12 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply
  • 18:11 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply
  • 18:11 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-upload_ulsfo - 7.1.1-2~bpo13+wmf3 ()
  • 18:11 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-text_ulsfo - 7.1.1-2~bpo13+wmf3 ()
  • 18:10 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply
  • 18:09 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply
  • 18:08 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply
  • 18:06 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
  • 18:06 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
  • 18:04 taavi@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: locked for re-pooling codfw for read traffic, contact SRE for equestions (duration: 109m 23s)
  • 18:04 cdobbins@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ncredir5004.eqsin.wmnet with OS trixie
  • 18:02 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
  • 18:02 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
  • 18:01 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1009.eqiad.wmnet
  • 18:01 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1008.eqiad.wmnet
  • 18:01 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1008.eqiad.wmnet
  • 17:59 cdanis@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool codfw [reason: no reason specified, no task ID specified]
  • 17:59 cdanis@cumin1004: START - Cookbook sre.dns.admin DNS admin: pool codfw [reason: no reason specified, no task ID specified]
  • 17:58 hnowlan@cumin1004: END (FAIL) - Cookbook sre.switchdc.mediawiki.00-optional-warmup-caches (exit_code=99) for datacenter switchover from eqiad to codfw
  • 17:54 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1008.eqiad.wmnet
  • 17:54 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1008.eqiad.wmnet
  • 17:54 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1007.eqiad.wmnet
  • 17:54 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1007.eqiad.wmnet
  • 17:52 cdanis@cumin1004: conftool action : set/pooled=false; selector: name=codfw,dnsdisc=mwdebug.*
  • 17:52 swfrench@cumin1004: conftool action : set/pooled=false; selector: dnsdisc=mwdebug.*,name=codfw
  • 17:49 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=mw-.*-ro
  • 17:47 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=apus
  • 17:47 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=mwdebug.*
  • 17:47 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1007.eqiad.wmnet
  • 17:44 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=swift
  • 17:42 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=config-master|device-analytics|echostore|helm-charts|k8s-ingress-wikikube-ro|linkrecommendation|mathoid|restbase|restbase-async|rest-gateway-ro|mobileapps|mwdebug.*|push-notifications|recommendation-api|releases|wikifeeds
  • 17:38 dzahn@cumin2003: START - Cookbook sre.hosts.reimage for host zuul1005.eqiad.wmnet with OS trixie
  • 17:37 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-upload_magru and not P{cp7011.magru.wmnet} and A:cp - 7.1.1-2~bpo13+wmf3 ()
  • 17:37 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-text_magru and not P{cp7001.magru.wmnet} and A:cp - 7.1.1-2~bpo13+wmf3 ()
  • 17:34 dzahn@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host zuul1005.eqiad.wmnet with OS trixie
  • 17:32 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=citoid|zotero
  • 17:30 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=apertium|schema|termbox|proton|cxserver
  • 17:22 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir5004.eqsin.wmnet with OS trixie
  • 17:19 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=thumbor
  • 17:18 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=shellbox.*
  • 17:17 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=urldownloader
  • 17:17 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1007.eqiad.wmnet
  • 17:17 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1006.eqiad.wmnet
  • 17:17 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1006.eqiad.wmnet
  • 17:10 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1006.eqiad.wmnet
  • 17:05 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir1001.*
  • 16:55 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
  • 16:55 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
  • 16:54 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
  • 16:54 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
  • 16:49 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=mw-web-next-ro
  • 16:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1006.eqiad.wmnet
  • 16:40 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1005.eqiad.wmnet
  • 16:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1005.eqiad.wmnet
  • 16:40 dzahn@cumin2003: START - Cookbook sre.hosts.reimage for host zuul1005.eqiad.wmnet with OS trixie
  • 16:37 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=mw-web-ro
  • 16:33 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1005.eqiad.wmnet
  • 16:33 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=mw-api-int-ro
  • 16:33 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir1001.eqiad.wmnet with OS trixie
  • 16:23 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply
  • 16:23 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply
  • 16:20 hnowlan@cumin1004: START - Cookbook sre.switchdc.mediawiki.00-optional-warmup-caches for datacenter switchover from eqiad to codfw
  • 16:19 hnowlan@cumin1004: END (FAIL) - Cookbook sre.switchdc.mediawiki.00-optional-warmup-caches (exit_code=99) for datacenter switchover from eqiad to codfw
  • 16:15 taavi@deploy1003: Locking from deployment [ALL REPOSITORIES]: locked for re-pooling codfw for read traffic, contact SRE for equestions
  • 16:14 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir1001.eqiad.wmnet with reason: host reimage
  • 16:14 dreamyjazz@deploy1003: Finished scap sync-world: Backport for AbuseReview: Enable on enwiki (T439149), Sync wmf/1.47.0-wmf.20 with wmf/1.47.0-wmf.21 for vandalism alpha (T438467) (duration: 33m 52s)
  • 16:08 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir1001.eqiad.wmnet with reason: host reimage
  • 16:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1005.eqiad.wmnet
  • 16:03 swfrench@cumin1004: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) restart A:liberica-ulsfo
  • 16:03 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1004.eqiad.wmnet
  • 16:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1004.eqiad.wmnet
  • 16:01 hnowlan@cumin1004: START - Cookbook sre.switchdc.mediawiki.00-optional-warmup-caches for datacenter switchover from eqiad to codfw
  • 16:01 swfrench@cumin1004: START - Cookbook sre.loadbalancer.upgrade restart A:liberica-ulsfo
  • 16:01 dreamyjazz@deploy1003: kharlan, dreamyjazz: Continuing with deployment
  • 16:00 dreamyjazz@deploy1003: kharlan, dreamyjazz: Backport for AbuseReview: Enable on enwiki (T439149), Sync wmf/1.47.0-wmf.20 with wmf/1.47.0-wmf.21 for vandalism alpha (T438467) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 15:57 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
  • 15:57 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
  • 15:56 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1004.eqiad.wmnet
  • 15:53 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1017.eqiad.wmnet
  • 15:52 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir1001.eqiad.wmnet with OS trixie
  • 15:51 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir3005.*
  • 15:51 swfrench-wmf: begin rolling restarts of confds in eqsin, codfw, ulsfo to reflect etcd SRV record changes
  • 15:47 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1017.eqiad.wmnet
  • 15:40 dreamyjazz@deploy1003: Started scap sync-world: Backport for AbuseReview: Enable on enwiki (T439149), Sync wmf/1.47.0-wmf.20 with wmf/1.47.0-wmf.21 for vandalism alpha (T438467)
  • 15:35 vgutierrez@dns1004: END - running authdns-update
  • 15:33 vgutierrez@dns1004: START - running authdns-update
  • 15:32 dreamyjazz@deploy1003: Finished scap sync-world: Backport for EventMapper::fetchByPage: Allow filtering by type (T438031) (duration: 12m 33s)
  • 15:30 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
  • 15:30 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
  • 15:29 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
  • 15:27 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
  • 15:27 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
  • 15:26 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1004.eqiad.wmnet
  • 15:26 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1003.eqiad.wmnet
  • 15:26 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1003.eqiad.wmnet
  • 15:25 dreamyjazz@deploy1003: kharlan, dreamyjazz: Continuing with deployment
  • 15:24 dreamyjazz@deploy1003: kharlan, dreamyjazz: Backport for EventMapper::fetchByPage: Allow filtering by type (T438031) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 15:20 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1003.eqiad.wmnet
  • 15:20 dreamyjazz@deploy1003: Started scap sync-world: Backport for EventMapper::fetchByPage: Allow filtering by type (T438031)
  • 15:18 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir3005.esams.wmnet with OS trixie
  • 15:12 vgutierrez@puppetserver1001: conftool action : set/pooled=yes; selector: dc=codfw,cluster=dnsbox
  • 15:06 vgutierrez@dns1004: END - running authdns-update
  • 15:04 vgutierrez@dns1004: START - running authdns-update
  • 15:03 vgutierrez@puppetserver1001: conftool action : set/pooled=yes; selector: name=dns2.*,service=authdns-update
  • 14:59 kharlan@deploy1003: Finished scap sync-world: Backport for AbuseReview: Add local CheckUsers to vandalism alpha test (T438467), AbuseReview: Inidicate if the queue hides recent edits (T438235) (duration: 32m 20s)
  • 14:57 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
  • 14:54 dkertesz@cumin1004: conftool action : set/pooled=yes; selector: name=cp7011.*
  • 14:54 dkertesz@cumin1004: conftool action : set/pooled=yes; selector: name=cp7001.*
  • 14:54 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply
  • 14:53 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply
  • 14:53 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply
  • 14:53 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply
  • 14:51 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply
  • 14:51 dkertesz: repooling cp7001|7011 after successful testing (T343000)
  • 14:50 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply
  • 14:49 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1003.eqiad.wmnet
  • 14:49 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1002.eqiad.wmnet
  • 14:49 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1002.eqiad.wmnet
  • 14:47 kharlan@deploy1003: kharlan: Continuing with deployment
  • 14:46 kharlan@deploy1003: kharlan: Backport for AbuseReview: Add local CheckUsers to vandalism alpha test (T438467), AbuseReview: Inidicate if the queue hides recent edits (T438235) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 14:43 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir3005.esams.wmnet with reason: host reimage
  • 14:40 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1016.eqiad.wmnet
  • 14:40 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1015.eqiad.wmnet
  • 14:40 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1016.eqiad.wmnet
  • 14:40 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1015.eqiad.wmnet
  • 14:40 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=codfw,name=dse-k8s-worker1016.eqiad.wmnet
  • 14:40 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1002.eqiad.wmnet
  • 14:40 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=codfw,name=dse-k8s-worker1015.eqiad.wmnet
  • 14:39 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=codfw,name=dse-k8s-worker1016.eqiad.wmnet
  • 14:39 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=codfw,name=dse-k8s-worker1015.eqiad.wmnet
  • 14:39 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on P{cp[5025,5026].*} and A:cp - 3.2.23 upgrade (T438828)
  • 14:39 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir3005.esams.wmnet with reason: host reimage
  • 14:37 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1002.eqiad.wmnet
  • 14:37 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1001.eqiad.wmnet
  • 14:37 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1001.eqiad.wmnet
  • 14:34 dkertesz@cumin1004: conftool action : set/pooled=no; selector: name=cp7011.*
  • 14:33 dkertesz@cumin1004: conftool action : set/pooled=no; selector: name=cp7001.*
  • 14:32 dkertesz: depooling cp7001|7011 to apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/1344222 (context: https://phabricator.wikimedia.org/T343000)
  • 14:31 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1001.eqiad.wmnet
  • 14:30 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1001.eqiad.wmnet
  • 14:30 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P{dse-k8s-worker1*.eqiad.wmnet} and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad)
  • 14:27 kharlan@deploy1003: Started scap sync-world: Backport for AbuseReview: Add local CheckUsers to vandalism alpha test (T438467), AbuseReview: Inidicate if the queue hides recent edits (T438235)
  • 14:26 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P{dse-k8s-wdqs-test1001.eqiad.wmnet} and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad)
  • 14:26 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs-test1001.eqiad.wmnet
  • 14:26 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs-test1001.eqiad.wmnet
  • 14:22 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'.
  • 14:22 elukey: elukey@rdb2013:/srv/redis/appendonlydir$ sudo -u redis redis-check-aof --fix rdb2013-6380.aof.22039.incr.aof
  • 14:21 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'.
  • 14:21 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on P{cp[5025,5026].*} and A:cp - 3.2.23 upgrade (T438828)
  • 14:20 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs-test1001.eqiad.wmnet
  • 14:19 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs-test1001.eqiad.wmnet
  • 14:19 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P{dse-k8s-wdqs-test1001.eqiad.wmnet} and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad)
  • 14:17 moritzm: installing Bird security updates
  • 14:13 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P{dse-k8s-wdqs100[1-3].eqiad.wmnet} and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad)
  • 14:13 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1003.eqiad.wmnet
  • 14:13 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1003.eqiad.wmnet
  • 14:11 vgutierrez@cumin1004: END (FAIL) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=1) rolling upgrade of HAProxy on A:cp-text_eqsin and A:cp - 3.2.23 upgrade (T438828)
  • 14:09 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply
  • 14:09 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir3005.esams.wmnet with OS trixie
  • 14:08 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply
  • 14:07 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1003.eqiad.wmnet
  • 14:07 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir4004.*
  • 14:07 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1003.eqiad.wmnet
  • 14:07 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1002.eqiad.wmnet
  • 14:07 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1002.eqiad.wmnet
  • 14:07 urbanecm@deploy1003: Finished scap sync-world: Backport for fix(AccountSetup): ensure TestKitchen knows about new user in CentralAuth redirect (T436872) (duration: 12m 27s)
  • 14:05 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply
  • 14:05 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply
  • 14:03 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply
  • 14:01 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1002.eqiad.wmnet
  • 14:01 urbanecm@deploy1003: migr, urbanecm: Continuing with deployment
  • 14:01 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1002.eqiad.wmnet
  • 14:01 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1001.eqiad.wmnet
  • 14:01 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1001.eqiad.wmnet
  • 14:00 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir4004.ulsfo.wmnet with OS trixie
  • 13:58 urbanecm@deploy1003: migr, urbanecm: Backport for fix(AccountSetup): ensure TestKitchen knows about new user in CentralAuth redirect (T436872) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 13:55 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1001.eqiad.wmnet
  • 13:55 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply
  • 13:55 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1001.eqiad.wmnet
  • 13:55 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply
  • 13:55 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P{dse-k8s-wdqs100[1-3].eqiad.wmnet} and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad)
  • 13:54 urbanecm@deploy1003: Started scap sync-world: Backport for fix(AccountSetup): ensure TestKitchen knows about new user in CentralAuth redirect (T436872)
  • 13:40 awight@deploy1003: Finished scap sync-world: Backport for Fixes failing edge when page is missing and entity usage remain. Updating ReallyDoQuery to function like an inner join. (T437687) (duration: 10m 38s)
  • 13:39 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply
  • 13:39 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir4004.ulsfo.wmnet with reason: host reimage
  • 13:35 moritzm: installing nghttp2 security updates
  • 13:35 awight@deploy1003: awight: Continuing with deployment
  • 13:34 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir4004.ulsfo.wmnet with reason: host reimage
  • 13:33 awight@deploy1003: awight: Backport for Fixes failing edge when page is missing and entity usage remain. Updating ReallyDoQuery to function like an inner join. (T437687) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 13:29 awight@deploy1003: Started scap sync-world: Backport for Fixes failing edge when page is missing and entity usage remain. Updating ReallyDoQuery to function like an inner join. (T437687)
  • 13:26 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply
  • 13:26 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply
  • 13:26 elukey@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'sync'.
  • 13:26 elukey@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'sync'.
  • 13:25 elukey@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'sync'.
  • 13:25 elukey@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'sync'.
  • 13:25 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply
  • 13:24 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply
  • 13:18 mlitn@deploy1003: Finished scap sync-world: Backport for Instrument five-arm image carousel retest (T431362), Wire image carousel retest instrumentation (T431362), ThumbExtractor: trim nbsp and dangling colons from caption text (T435672), ThumbExtractor: exclude lead infobox images from the carousel (T438907) (duration: 12m 25s)
  • 13:13 mlitn@deploy1003: mfossati, mlitn: Continuing with deployment
  • 13:10 mlitn@deploy1003: mfossati, mlitn: Backport for Instrument five-arm image carousel retest (T431362), Wire image carousel retest instrumentation (T431362), ThumbExtractor: trim nbsp and dangling colons from caption text (T435672), ThumbExtractor: exclude lead infobox images from the carousel (T438907) synced to the testservers (see https://wiki
  • 13:08 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir4004.ulsfo.wmnet with OS trixie
  • 13:07 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply
  • 13:07 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply
  • 13:06 mlitn@deploy1003: Started scap sync-world: Backport for Instrument five-arm image carousel retest (T431362), Wire image carousel retest instrumentation (T431362), ThumbExtractor: trim nbsp and dangling colons from caption text (T435672), ThumbExtractor: exclude lead infobox images from the carousel (T438907)
  • 13:06 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply
  • 13:06 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply
  • 13:02 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply
  • 13:02 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply
  • 13:02 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_eqsin and A:cp - 3.2.23 upgrade (T438828)
  • 13:01 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=cp2059.*
  • 12:59 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply
  • 12:59 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply
  • 12:58 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply
  • 12:58 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply
  • 12:52 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply
  • 12:52 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply
  • 12:43 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply
  • 12:43 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply
  • 12:34 urbanecm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply
  • 12:34 urbanecm@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply
  • 12:04 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
  • 12:03 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
  • 11:21 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on P{cp[5031,5032].*} and A:cp - 3.2.23 upgrade (T438828)
  • 11:13 hnowlan: restarted restbase on restbase2029
  • 11:04 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on P{cp[5031,5032].*} and A:cp - 3.2.23 upgrade (T438828)
  • 10:50 hnowlan: deleting stuck mw-web pods in eqiad
  • 10:45 dreamyjazz@deploy1003: Finished scap sync-world: Backport for AbuseReview: Let specific users and suppressors see vandalism tag (T438860) (duration: 10m 09s)
  • 10:44 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply
  • 10:42 vgutierrez@cumin1004: END (FAIL) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=1) rolling upgrade of HAProxy on A:cp-upload_eqsin and A:cp - 3.2.23 upgrade (T438828)
  • 10:40 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment
  • 10:39 dreamyjazz@deploy1003: dreamyjazz: Backport for AbuseReview: Let specific users and suppressors see vandalism tag (T438860) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 10:36 filippo@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on cloudvirt1080.eqiad.wmnet with reason: provision
  • 10:35 dreamyjazz@deploy1003: Started scap sync-world: Backport for AbuseReview: Let specific users and suppressors see vandalism tag (T438860)
  • 10:34 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply
  • 10:32 dreamyjazz@deploy1003: Finished scap sync-world: Backport for WikimediaAntiAbuse: Enable likely vandalism classifier on testwiki (T438860) (duration: 10m 34s)
  • 10:29 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
  • 10:26 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment
  • 10:26 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
  • 10:25 dreamyjazz@deploy1003: dreamyjazz: Backport for WikimediaAntiAbuse: Enable likely vandalism classifier on testwiki (T438860) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 10:23 filippo@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on cloudvirt1079.eqiad.wmnet with reason: provision
  • 10:22 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply
  • 10:21 dreamyjazz@deploy1003: Started scap sync-world: Backport for WikimediaAntiAbuse: Enable likely vandalism classifier on testwiki (T438860)
  • 10:17 rzl@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: No deployments please, as we're still cleaning up from the codfw power incident T439010. Thursday UTC morning at the earliest, but please ask SRE oncall. (duration: 653m 55s)
  • 10:17 hnowlan@deploy1003: Forcefully removing global lock: Unlocking scap after restoration of power in codfw
  • 10:12 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply
  • 10:11 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-text_ulsfo and A:cp - 3.2.23 upgrade (T438828)
  • 10:08 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_eqsin and A:cp - 3.2.23 upgrade (T438828)
  • 10:03 moritzm: installing apr-util security updates
  • 09:46 moritzm: installing bind9 security updates (client-side tools/libs only)
  • 09:40 vgutierrez@puppetserver1001: conftool action : set/pooled=no; selector: name=cirrussearch1120.eqiad.wmnet
  • 09:27 ayounsi@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool drmrs [reason: switch upgrade, T437984]
  • 09:27 ayounsi@cumin1004: START - Cookbook sre.dns.admin DNS admin: pool drmrs [reason: switch upgrade, T437984]
  • 09:26 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.depool-rack (exit_code=0) with action 'pool' for drmrs rack B13
  • 09:25 ayounsi@cumin1004: START - Cookbook sre.network.depool-rack with action 'pool' for drmrs rack B13
  • 09:23 arthurtaylor@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikidata-query-gui: apply
  • 09:22 arthurtaylor@deploy1003: helmfile [codfw] START helmfile.d/services/wikidata-query-gui: apply
  • 09:22 arthurtaylor@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply
  • 09:22 arthurtaylor@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply
  • 09:21 arthurtaylor@deploy1003: helmfile [staging] DONE helmfile.d/services/wikidata-query-gui: apply
  • 09:21 arthurtaylor@deploy1003: helmfile [staging] START helmfile.d/services/wikidata-query-gui: apply
  • 09:10 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1016.eqiad.wmnet
  • 09:05 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_ulsfo and A:cp - 3.2.23 upgrade (T438828)
  • 09:04 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1016.eqiad.wmnet
  • 09:01 XioNoX: asw1-b13-drmrs> request system reboot - T437984
  • 09:00 jelto@cumin1004: END (PASS) - Cookbook sre.gitlab.reboot-runner (exit_code=0) rolling reboot on A:gitlab-runner
  • 09:00 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.depool-rack (exit_code=0) with action 'depool' for drmrs rack B13
  • 08:59 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudvirt1078.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 08:58 moritzm: installing node-lodash security updates
  • 08:56 ayounsi@cumin1004: START - Cookbook sre.network.depool-rack with action 'depool' for drmrs rack B13
  • 08:55 filippo@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on cloudvirt1078.eqiad.wmnet with reason: provision
  • 08:54 filippo@cumin1004: START - Cookbook sre.hosts.provision for host cloudvirt1078.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 08:49 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.depool-rack (exit_code=0) with action 'pool' for drmrs rack B12
  • 08:47 ayounsi@cumin1004: START - Cookbook sre.network.depool-rack with action 'pool' for drmrs rack B12
  • 08:46 ayounsi@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 19 hosts with reason: Switches upgrade
  • 08:46 moritzm: uploaded debuerreotype 0.15-1.1+wmf13u1 to component/main from trixie-wikimedia T438866
  • 08:45 ayounsi@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for asw1-b12-drmrs,asw1-b12-drmrs IPv6,asw1-b12-drmrs.mgmt
  • 08:45 ayounsi@cumin1004: START - Cookbook sre.hosts.remove-downtime for asw1-b12-drmrs,asw1-b12-drmrs IPv6,asw1-b12-drmrs.mgmt
  • 08:45 ayounsi@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on asw1-b13-drmrs,asw1-b13-drmrs IPv6,asw1-b13-drmrs.mgmt with reason: Switch upgrade
  • 08:37 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P{dse-k8s-worker1015.eqiad.wmnet} and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad)
  • 08:37 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host dse-k8s-worker1015.eqiad.wmnet
  • 08:37 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1015.eqiad.wmnet
  • 08:31 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1015.eqiad.wmnet
  • 08:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1015.eqiad.wmnet
  • 08:31 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P{dse-k8s-worker1015.eqiad.wmnet} and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad)
  • 08:22 XioNoX: asw1-b12-drmrs> request system reboot - T437984
  • 08:20 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.depool-rack (exit_code=0) with action 'depool' for drmrs rack B12
  • 08:13 ayounsi@cumin1004: START - Cookbook sre.network.depool-rack with action 'depool' for drmrs rack B12
  • 08:06 jelto@cumin1004: START - Cookbook sre.gitlab.reboot-runner rolling reboot on A:gitlab-runner
  • 08:02 ayounsi@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on asw1-b12-drmrs,asw1-b12-drmrs IPv6,asw1-b12-drmrs.mgmt with reason: Switch upgrade
  • 07:53 ayounsi@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 20 hosts with reason: Switches upgrade
  • 07:52 ayounsi@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool drmrs [reason: switch upgrade, T437984]
  • 07:52 ayounsi@cumin1004: START - Cookbook sre.dns.admin DNS admin: depool drmrs [reason: switch upgrade, T437984]
  • 07:48 jelto@cumin1004: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: version upgrade
  • 07:19 jelto@cumin1004: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: version upgrade
  • 07:16 jelto@cumin1004: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab2002.wikimedia.org with reason: version upgrade
  • 07:06 jelto@cumin1004: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab2002.wikimedia.org with reason: version upgrade
  • 07:02 jelto@cumin1004: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1003.wikimedia.org with reason: version upgrade
  • 06:51 jelto@cumin1004: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: version upgrade
  • 06:41 kart_: staging: Update machinetranslation/MinT to 2026-09-21-112314-production (T437213)
  • 06:41 kartik@deploy1003: helmfile [staging] DONE helmfile.d/services/machinetranslation: apply
  • 06:39 kart_: staging: Update machinetranslation/MinT to 2026-09-21-112314-production
  • 06:38 kartik@deploy1003: helmfile [staging] START helmfile.d/services/machinetranslation: apply
  • 06:07 ayounsi@cumin1004: END (FAIL) - Cookbook sre.network.tls (exit_code=99) for network device lsw1-e5-codfw
  • 06:06 ayounsi@cumin1004: START - Cookbook sre.network.tls for network device lsw1-e5-codfw
  • 05:07 ryankemper@cumin2003: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster search_codfw: Restart codfw following today's power incident to ensure we return to our full expected state - ryankemper@cumin2003 - T439010
  • 01:21 ryankemper@cumin2003: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster search_codfw: Restart codfw following today's power incident to ensure we return to our full expected state - ryankemper@cumin2003 - T439010
  • 01:19 ryankemper: [Cirrus] Reverted `node_concurrent_recoveries` to 5 from 10, now that we're back to green
  • 01:16 ryankemper: [Cirrus] With the restart of `cirrussearch2115`, the codfw cluster has officially reached green status!!! Still working on full verification, but we're almost done here
  • 01:14 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2115.codfw.wmnet with reason: Codfw survivor recovery on 2115; temporary chi red expected (T439010)
  • 01:11 brett@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp2059.codfw.wmnet with reason: failing services but not in service yet
  • 01:10 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2109.codfw.wmnet with reason: Codfw survivor recovery on 2109; temporary chi red expected (T439010)
  • 01:04 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2104.codfw.wmnet with reason: Codfw survivor recovery on 2104; temporary chi red expected (T439010)
  • 01:03 ryankemper: [Cirrus] grr, I'd missed some hosts. restarting the last few dangling ones, we're really close to back to green, prob 3-ish more hosts
  • 00:40 ryankemper: [Cirrus] Great news, we briefly dipped red (same as previous restarts) but went back to yellow almost immediately. AFAICT election went fine, still checking though
  • 00:38 ryankemper: [Cirrus] Preparing to restart cirrussearch2084 (active cluster manager). With luck, this should restore updater availability (and general cluster green status, after some reshuffling)
  • 00:35 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on 55 hosts with reason: Codfw chi elected-manager recovery on 2084; expected brief failover and red state (T439010)
  • 00:10 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp7011.*
  • 00:05 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on P{cp7011.magru.wmnet} and A:cp - 7.1.1-2~bpo13+wmf3 ()
  • 00:00 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P{cp7011.magru.wmnet} and A:cp - 7.1.1-2~bpo13+wmf3 ()

2026-09-23

  • 23:58 dzahn@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host zuul1005.eqiad.wmnet with OS trixie
  • 23:56 brett: Switching acme-chief primary from codfw to eqiad - T439010
  • 23:54 brett@cumin1004: END (FAIL) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=1) rolling upgrade of Varnish on P{cp7011.magru.wmnet} and A:cp - 7.1.1-2~bpo13+wmf3 ()
  • 23:49 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P{cp7011.magru.wmnet} and A:cp - 7.1.1-2~bpo13+wmf3 ()
  • 23:48 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on P{cp7001.magru.wmnet} and A:cp - 7.1.1-2~bpo13+wmf3 ()
  • 23:48 ryankemper: [Cirrus] Every host except 2084, which is the current elected chi master, has now been restarted, and shard recoveries healed accordingly. AFAICT we will not be able to revive the updater until we restart this host. Pausing for a few mins to mull things over and get my bearings though, because this restart would be higher-touch than the previous ones
  • 23:38 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2108.codfw.wmnet with reason: Codfw survivor recovery on 2108; sequential chi and psi restarts (T439010)
  • 23:38 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P{cp7001.magru.wmnet} and A:cp - 7.1.1-2~bpo13+wmf3 ()
  • 23:35 brett: import varnish 7.1.1-2~bpo13+wmf3 into trixie-wikimedia (T438293)
  • 23:34 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2107.codfw.wmnet with reason: Codfw survivor recovery on 2107; sequential chi and psi restarts (T439010)
  • 23:27 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2085.codfw.wmnet with reason: Codfw survivor recovery on 2085; sequential chi and psi restarts (T439010)
  • 23:23 rzl@deploy1003: Locking from deployment [ALL REPOSITORIES]: No deployments please, as we're still cleaning up from the codfw power incident T439010. Thursday UTC morning at the earliest, but please ask SRE oncall.
  • 23:23 rzl@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: incident recovery in progress T439010 (duration: 121m 40s)
  • 23:20 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2072.codfw.wmnet with reason: Codfw survivor recovery on 2072; sequential chi and psi restarts (T439010)
  • 23:09 ryankemper: [Cirrus] rolling cirrussearch2086 next
  • 23:08 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2086.codfw.wmnet with reason: Codfw survivor recovery on 2086; sequential chi and omega restarts (T439010)
  • 23:01 ryankemper: [Cirrus] Doing cirrussearch2114 next
  • 22:59 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2114.codfw.wmnet with reason: Codfw survivor recovery on 2114; sequential chi and omega restarts (T439010)
  • 22:44 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2106.codfw.wmnet with reason: Codfw chi survivor recovery on 2106; temporary red expected (T439010)
  • 22:29 ryankemper: [Cirrus] proceeding with manual restart of cirrussearch2105; red status expected, hopefully brief but we'll see
  • 22:28 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2105.codfw.wmnet with reason: Codfw chi recovery canary on 2105; temporary service interruption expected (T439010)
  • 22:24 ryankemper: [Cirrus] s/expected/expect
  • 22:23 ryankemper: [Cirrus] Alright, I'm getting increasingly convinced that there's no way to restore healthy cluster state without inevitably having to restart sole-shard-holder hosts, which will put the cluster into red status. going to start with just `cirrussearch2105`; I expected red status. silencing alerts first so I don't blow out the channel
  • 22:08 ryankemper: [Cirrus] (to be clear the cluster is not serving live traffic, but if I can avoid red I will)
  • 22:08 ryankemper: [Cirrus] updater still failing in codfw cirrussearch; i've restarted the directly-impacted hosts but not the others. some bulk updates appear to be getting rejected, going to do some targeted restarts and assess impact before considering a broader operation. first up is `cirrussearch2071.codfw.wmnet` which is not the sole holder of any shards therefore should not plunge the cluster into red status
  • 21:49 dzahn@cumin2003: START - Cookbook sre.hosts.reimage for host zuul1005.eqiad.wmnet with OS trixie
  • 21:22 rzl@deploy1003: Locking from deployment [ALL REPOSITORIES]: incident recovery in progress T439010
  • 21:22 rzl@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: incident recovery in progress T439010 (duration: 51m 29s)
  • 21:21 cdobbins@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ncredir5004.eqsin.wmnet with OS trixie
  • 21:18 Emperor: ceph mgr fail on apus-be2005
  • 21:18 Emperor: reset-failed then restart ceph-mon on moss-be2003
  • 21:08 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db[2160,2235].codfw.wmnet with reason: needs fixing
  • 21:08 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db[2160,2234].codfw.wmnet with reason: needs fixing
  • 21:07 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db[2160,2233].codfw.wmnet with reason: needs fixing
  • 21:07 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db[2160,2232].codfw.wmnet with reason: needs fixing
  • 20:57 ryankemper: [Cirrus] cirrussearch codfw back to yellow status. active shard pct = 94.51%
  • 20:55 ryankemper: [Cirrus] Bump codfw cirrussearch shard recoveries from 5 to 10; cluster not serving live traffic so I'm hoping we have headroom to recover faster
  • 20:49 swfrench@dns1004: END - running authdns-update
  • 20:46 swfrench@dns1004: START - running authdns-update
  • 20:41 ryankemper: [Cirrus] Been restarting all impacted codfw opensearch hosts one at a time (they didn't rejoin the cluster naturally)
  • 20:39 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir5004.eqsin.wmnet with OS trixie
  • 20:38 cdobbins@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ncredir5004.eqsin.wmnet with OS trixie
  • 20:30 rzl@deploy1003: Locking from deployment [ALL REPOSITORIES]: incident recovery in progress T439010
  • 20:27 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
  • 20:27 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
  • 20:06 dzahn@dns1004: END - running authdns-update
  • 20:03 dzahn@dns1004: START - running authdns-update
  • 19:52 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir5004.eqsin.wmnet with OS trixie
  • 19:34 sukhe@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp2059.codfw.wmnet with OS trixie
  • 19:33 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply
  • 19:33 volans: rebooting arclamp2001.codfw.wmnet
  • 19:32 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply
  • 19:20 sukhe@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on 979 hosts with reason: power is still coming back on
  • 19:17 taavi@dns1004: END - running authdns-update
  • 19:14 taavi@dns1004: START - running authdns-update
  • 19:10 taavi@cumin1004: END (PASS) - Cookbook sre.gerrit.read-only-toggle (exit_code=0) from gerrit1003.wikimedia.org
  • 19:10 taavi@cumin1004: START - Cookbook sre.gerrit.read-only-toggle from gerrit1003.wikimedia.org
  • 19:10 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir6001.*
  • 19:08 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: dc=codfw,cluster=dnsbox,service=authdns-update
  • 18:59 sukhe@cumin1004: DONE (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 6:00:00 on 980 hosts with reason: power is still coming back on
  • 18:58 taavi@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) gerrit.discovery.wmnet on all recursors
  • 18:58 taavi@cumin1004: START - Cookbook sre.dns.wipe-cache gerrit.discovery.wmnet on all recursors
  • 18:50 taavi@cumin1004: END (PASS) - Cookbook sre.gerrit.localbackup (exit_code=0) Prepare local backup on: gerrit2003.wikimedia.org
  • 18:45 sukhe@dns1004: END - running authdns-update
  • 18:43 sukhe@dns1004: START - running authdns-update
  • 18:43 taavi@cumin1004: START - Cookbook sre.gerrit.localbackup Prepare local backup on: gerrit2003.wikimedia.org
  • 18:42 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: dc=codfw,cluster=dnsbox,service=authdns-update
  • 18:42 dzahn@cumin2003: END (FAIL) - Cookbook sre.gerrit.localbackup (exit_code=99) Prepare local backup on: gerrit2003.wikimedia.org
  • 18:42 dzahn@cumin2003: START - Cookbook sre.gerrit.localbackup Prepare local backup on: gerrit2003.wikimedia.org
  • 18:40 dzahn@cumin2003: END (FAIL) - Cookbook sre.gerrit.localbackup (exit_code=99) Prepare local backup on: gerrit2003.wikimedia.org
  • 18:40 dzahn@cumin2003: START - Cookbook sre.gerrit.localbackup Prepare local backup on: gerrit2003.wikimedia.org
  • 18:40 dzahn@cumin2003: END (FAIL) - Cookbook sre.gerrit.localbackup (exit_code=99) Prepare local backup on: gerrit2003.wikimedia.org
  • 18:40 dzahn@cumin2003: START - Cookbook sre.gerrit.localbackup Prepare local backup on: gerrit2003.wikimedia.org
  • 18:40 taavi@cumin1004: END (PASS) - Cookbook sre.gerrit.localbackup (exit_code=0) Prepare local backup on: gerrit1003.wikimedia.org
  • 18:38 cdanis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) _etcd-client-ssl._tcp.eqsin.wmnet _etcd-client-ssl._tcp.ulsfo.wmnet _etcd-client-ssl._tcp.codfw.wmnet on all recursors
  • 18:38 cdanis@cumin1004: START - Cookbook sre.dns.wipe-cache _etcd-client-ssl._tcp.eqsin.wmnet _etcd-client-ssl._tcp.ulsfo.wmnet _etcd-client-ssl._tcp.codfw.wmnet on all recursors
  • 18:36 taavi@cumin1004: END (PASS) - Cookbook sre.gerrit.read-only-toggle (exit_code=0) from gerrit1003.wikimedia.org
  • 18:36 taavi@cumin1004: START - Cookbook sre.gerrit.read-only-toggle from gerrit1003.wikimedia.org
  • 18:36 taavi@cumin1004: END (PASS) - Cookbook sre.gerrit.read-only-toggle (exit_code=0) from gerrit2003.wikimedia.org
  • 18:36 taavi@cumin1004: START - Cookbook sre.gerrit.read-only-toggle from gerrit2003.wikimedia.org
  • 18:30 taavi@cumin1004: START - Cookbook sre.gerrit.localbackup Prepare local backup on: gerrit1003.wikimedia.org
  • 18:29 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
  • 18:28 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
  • 18:14 vriley@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host zuul1005.eqiad.wmnet with OS trixie
  • 18:08 sukhe@cumin1004: END (FAIL) - Cookbook sre.dns.wipe-cache (exit_code=99) idp.wikimedia.org on all recursors
  • 18:08 sukhe@cumin1004: START - Cookbook sre.dns.wipe-cache idp.wikimedia.org on all recursors
  • 18:05 cdanis@cumin1004: END (FAIL) - Cookbook sre.dns.wipe-cache (exit_code=99) _etcd-client-ssl._tcp.eqsin.wmnet on all recursors
  • 18:05 cdanis@cumin1004: START - Cookbook sre.dns.wipe-cache _etcd-client-ssl._tcp.eqsin.wmnet on all recursors
  • 18:03 cdanis@cumin1004: END (FAIL) - Cookbook sre.dns.wipe-cache (exit_code=99) _etcd-client-ssl._tcp.eqsin.wmnet on all recursors
  • 18:03 cdanis@cumin1004: START - Cookbook sre.dns.wipe-cache _etcd-client-ssl._tcp.eqsin.wmnet on all recursors
  • 18:02 cdanis@cumin1004: END (FAIL) - Cookbook sre.dns.wipe-cache (exit_code=99) _etcd-client-ssl._tcp.ulsfo.wmnet on all recursors
  • 18:02 cdanis@cumin1004: START - Cookbook sre.dns.wipe-cache _etcd-client-ssl._tcp.ulsfo.wmnet on all recursors
  • 18:01 cdanis@dns1005: END - running authdns-update
  • 17:58 cdanis@dns1005: START - running authdns-update
  • 17:57 vriley@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on zuul1005.eqiad.wmnet with reason: host reimage
  • 17:54 taavi@dns1004: END - running authdns-update
  • 17:53 vriley@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on zuul1005.eqiad.wmnet with reason: host reimage
  • 17:51 taavi@dns1004: START - running authdns-update
  • 17:46 taavi@dns1004: END - running authdns-update
  • 17:43 taavi@dns1004: START - running authdns-update
  • 17:37 vriley@cumin1004: START - Cookbook sre.hosts.reimage for host zuul1005.eqiad.wmnet with OS trixie
  • 17:35 rzl@cumin1004: START - Cookbook sre.discovery.datacenter pool all active/active services in eqiad: maintenance - T439010
  • 17:35 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir6001.drmrs.wmnet with OS trixie
  • 17:35 cdanis@cumin1004: END (FAIL) - Cookbook sre.dns.admin (exit_code=99) DNS admin: depool codfw [reason: no reason specified, no task ID specified]
  • 17:35 cdanis@cumin1004: START - Cookbook sre.dns.admin DNS admin: depool codfw [reason: no reason specified, no task ID specified]
  • 17:24 sukhe@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool codfw [reason: no reason specified, no task ID specified]
  • 17:23 sukhe@cumin1004: START - Cookbook sre.dns.admin DNS admin: depool codfw [reason: no reason specified, no task ID specified]
  • 17:21 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply
  • 17:21 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply
  • 17:18 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply
  • 17:17 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply
  • 17:16 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply
  • 17:14 sukhe@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp2059.codfw.wmnet with reason: host reimage
  • 17:11 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: name=cp2049.codfw.wmnet
  • 17:11 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: name=cp2049
  • 17:10 sukhe@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on cp2059.codfw.wmnet with reason: host reimage
  • 17:07 mutante: cloudcontrol2005-dev, cloudcontrol2006-dev, cloudcontrol2010-dev: restart zookeeper, enabled logging (/var/log/zookeeper/zookeeper.log) after gerrit:1342354
  • 17:02 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir6001.drmrs.wmnet with reason: host reimage
  • 16:59 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir6001.drmrs.wmnet with reason: host reimage
  • 16:51 sukhe@cumin1004: START - Cookbook sre.hosts.reimage for host cp2059.codfw.wmnet with OS trixie
  • 16:51 sukhe@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp2059.codfw.wmnet with OS trixie
  • 16:48 sukhe@cumin1004: START - Cookbook sre.hosts.reimage for host cp2059.codfw.wmnet with OS trixie
  • 16:39 sukhe@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp2059.codfw.wmnet with OS trixie
  • 16:35 dzahn@cumin2003: START - Cookbook sre.hosts.reimage for host zuul1005.eqiad.wmnet with OS trixie
  • 16:34 dzahn@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host zuul1005.eqiad.wmnet with OS trixie
  • 16:30 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply
  • 16:29 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir6001.drmrs.wmnet with OS trixie
  • 16:10 sukhe@cumin1004: START - Cookbook sre.hosts.reimage for host cp2059.codfw.wmnet with OS trixie
  • 16:10 sukhe@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp2059.codfw.wmnet with OS trixie
  • 15:55 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-upload_ulsfo and A:cp - 3.2.23 upgrade (T438828)
  • 15:54 moritzm: installing cjose security updates
  • 15:54 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir7004.*
  • 15:53 dancy@deploy1003: Finished scap sync-world: testing (duration: 07m 06s)
  • 15:52 sukhe@cumin1004: START - Cookbook sre.hosts.reimage for host cp2059.codfw.wmnet with OS trixie
  • 15:52 sukhe@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp2059.codfw.wmnet with OS trixie
  • 15:51 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply
  • 15:50 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply
  • 15:50 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply
  • 15:46 dancy@deploy1003: Started scap sync-world: testing
  • 15:43 sukhe@cumin1004: START - Cookbook sre.hosts.reimage for host cp2059.codfw.wmnet with OS trixie
  • 15:42 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir7004.magru.wmnet with OS trixie
  • 15:42 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply
  • 15:41 sukhe: homer "lsw1-e4-codfw.*" commit 'pending from cookbook'
  • 15:41 Emperor: rclone copy --no-update-modtime --checksum --config /etc/swift/rclone.conf 'eqiad:wikipedia-commons-local-public.c7/c/c7/Kamāl_al-Dīn_Ḥusayn_b._ʿAlī_Bayhaqī_Sabzavārī_Vā‛iẓ_Kāšifī_._Anvār-i_Suhaylī_-_btv1b10515885n_(248_of_580).jpg' codfw:wikipedia-commons-local-public.c7/c/c7 T438961
  • 15:39 sukhe@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from sretest2013 to cp2059
  • 15:38 sukhe@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cp2059
  • 15:38 sukhe@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host cp2059
  • 15:38 sukhe@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cp2059 on all recursors
  • 15:38 Emperor: rclone copy --no-update-modtime --checksum --config /etc/swift/rclone.conf 'eqiad:wikipedia-commons-local-public.a9/a/a9/Ğāmi‛_al-tavārīḫ._Rašīd_al-Dīn_Fazl-ullāh_Hamadānī_-_btv1b8427170s_(182_of_597).jpg' codfw:wikipedia-commons-local-public.a9/a/a9/ T438961
  • 15:38 sukhe@cumin1004: START - Cookbook sre.dns.wipe-cache cp2059 on all recursors
  • 15:38 sukhe@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 15:38 sukhe@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming sretest2013 to cp2059 - sukhe@cumin1004"
  • 15:37 sukhe@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming sretest2013 to cp2059 - sukhe@cumin1004"
  • 15:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
  • 15:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
  • 15:35 Emperor: rclone copy --no-update-modtime --checksum --config /etc/swift/rclone.conf 'eqiad:wikipedia-commons-local-public.4d/4/4d/Kamāl_al-Dīn_Ḥusayn_b._ʿAlī_Bayhaqī_Sabzavārī_Vā‛iẓ_Kāšifī_._Anvār-i_Suhaylī_-_btv1b10515885n_(142_of_580).jpg' codfw:wikipedia-commons-local-public.4d/4/4d T438961
  • 15:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
  • 15:35 mutante: zuul1005 - reimage - should not have had nftables on it before T438786
  • 15:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
  • 15:34 dzahn@cumin2003: START - Cookbook sre.hosts.reimage for host zuul1005.eqiad.wmnet with OS trixie
  • 15:34 sukhe@cumin1004: START - Cookbook sre.dns.netbox
  • 15:33 sukhe@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059
  • 15:32 Emperor: rclone copy --no-update-modtime --checksum --config /etc/swift/rclone.conf 'eqiad:wikipedia-commons-local-public.41/4/41/ĞAVĀMI‛_al-ḤIKĀYĀT_VA_LAVĀMI‛_al-RIVĀYĀT._Sadīd_al-Dīn_Muḥ._b._Muḥ._b._Yaḥyà_‛Awfī_Buhārī_Ḥanafī._-_btv1b525129105_(033_of_524).jpg' codfw:wikipedia-commons-local-public.41/4/41 T438961
  • 15:23 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply
  • 15:23 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_ulsfo and A:cp - 3.2.23 upgrade (T438828)
  • 15:21 sukhe@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest2013.codfw.wmnet with OS trixie
  • 15:21 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mobileapps: apply
  • 15:21 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mobileapps: apply
  • 15:20 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mobileapps: apply
  • 15:20 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
  • 15:19 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
  • 15:18 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir7004.magru.wmnet with reason: host reimage
  • 15:14 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir7004.magru.wmnet with reason: host reimage
  • 15:12 jayme@deploy1003: conftool action : set/pooled=true; selector: dnsdisc=mw-web-ro,name=eqiad
  • 15:12 jayme@deploy1003: conftool action : set/pooled=true; selector: dnsdisc=mw-web-next-ro,name=eqiad
  • 15:12 moritzm: removed buster-wikimedia and all related components from apt.wikimedia.org following the merge of https://gerrit.wikimedia.org/r/c/operations/puppet/+/1247618
  • 15:06 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-upload_magru and not P{cp[7010,7016].*} and A:cp - 3.2.23 upgrade (T438828)
  • 15:02 dancy@deploy1003: Installation of scap version "4.292.0" completed for 3 hosts
  • 15:02 sukhe@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2013.codfw.wmnet with reason: host reimage
  • 15:02 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply
  • 15:01 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply
  • 15:01 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply
  • 15:01 jayme@deploy1003: conftool action : set/pooled=false; selector: dnsdisc=mw-web-next-ro,name=eqiad
  • 15:01 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply
  • 15:01 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply
  • 15:00 dancy@deploy1003: Installing scap version "4.292.0" for 3 host(s)
  • 15:00 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/wikifeeds: apply
  • 14:58 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply
  • 14:58 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply
  • 14:57 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply
  • 14:57 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply
  • 14:57 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply
  • 14:57 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/wikifeeds: apply
  • 14:56 sukhe@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2013.codfw.wmnet with reason: host reimage
  • 14:55 jayme@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) check mw-web-ro: maintenance
  • 14:55 jayme@cumin1004: START - Cookbook sre.discovery.service-route check mw-web-ro: maintenance
  • 14:55 jayme@cumin1004: END (FAIL) - Cookbook sre.discovery.service-route (exit_code=99) depool mw-web-ro in eqiad: maintenance
  • 14:55 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from codfw to eqiad for section test-s4
  • 14:54 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.finalize for the switch from codfw to eqiad for section test-s4
  • 14:54 jayme@cumin1004: START - Cookbook sre.discovery.service-route depool mw-web-ro in eqiad: maintenance
  • 14:54 jayme@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) check mw-web-ro: maintenance
  • 14:54 jayme@cumin1004: START - Cookbook sre.discovery.service-route check mw-web-ro: maintenance
  • 14:53 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from codfw to eqiad for section test-s4
  • 14:53 gengh@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
  • 14:53 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from codfw to eqiad for section test-s4
  • 14:47 gengh@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
  • 14:47 gengh@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
  • 14:47 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from eqiad to codfw for section test-s4
  • 14:46 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.finalize for the switch from eqiad to codfw for section test-s4
  • 14:45 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section test-s4
  • 14:45 gengh@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
  • 14:45 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
  • 14:44 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
  • 14:44 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section test-s4
  • 14:43 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from codfw to eqiad for section test-s4
  • 14:43 gengh@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
  • 14:42 gengh@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
  • 14:42 aqu@deploy1003: Finished deploy [analytics/refinery@58c9356] (thin): Regular analytics weekly train THIN [analytics/refinery@58c93566] (duration: 00m 12s)
  • 14:42 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from codfw to eqiad for section test-s4
  • 14:42 aqu@deploy1003: Started deploy [analytics/refinery@58c9356] (thin): Regular analytics weekly train THIN [analytics/refinery@58c93566]
  • 14:42 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir7004.magru.wmnet with OS trixie
  • 14:40 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from codfw to eqiad for section test-s4
  • 14:40 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.finalize for the switch from codfw to eqiad for section test-s4
  • 14:39 moritzm: upload debuerreotype 0.15-1.1+wmf13u1 to component/main from trixie-wikimedia T438866
  • 14:38 gengh@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
  • 14:38 gengh@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
  • 14:37 gengh@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
  • 14:37 urbanecm@deploy1003: Finished scap sync-world: Backport for feat(AddLink): Do not resuggest an already reviewed page (T429417), feat(AddLink): Do not resuggest an already reviewed page (T429417) (duration: 14m 34s)
  • 14:37 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_magru and not P{cp[7010,7016].*} and A:cp - 3.2.23 upgrade (T438828)
  • 14:37 gengh@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
  • 14:36 gengh@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
  • 14:36 gengh@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
  • 14:36 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-text_magru and not P{cp[7010,7016].*} and A:cp - 3.2.23 upgrade (T438828)
  • 14:28 sukhe@cumin1004: START - Cookbook sre.hosts.reimage for host sretest2013.codfw.wmnet with OS trixie
  • 14:28 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir1002.*
  • 14:26 gengh@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
  • 14:26 gengh@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
  • 14:24 gengh@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
  • 14:23 gengh@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
  • 14:23 urbanecm@deploy1003: Started scap sync-world: Backport for feat(AddLink): Do not resuggest an already reviewed page (T429417), feat(AddLink): Do not resuggest an already reviewed page (T429417)
  • 14:17 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir1002.eqiad.wmnet with OS trixie
  • 14:10 gengh@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
  • 14:09 gengh@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
  • 14:07 ebernhardson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search: apply
  • 14:07 ebernhardson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-semantic-search: apply
  • 13:58 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir1002.eqiad.wmnet with reason: host reimage
  • 13:56 sukhe@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2013.codfw.wmnet with OS trixie
  • 13:53 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir1002.eqiad.wmnet with reason: host reimage
  • 13:38 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_magru and not P{cp[7010,7016].*} and A:cp - 3.2.23 upgrade (T438828)
  • 13:37 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P{dse-k8s-wdqs-test1001.eqiad.wmnet} and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad)
  • 13:37 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs-test1001.eqiad.wmnet
  • 13:37 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs-test1001.eqiad.wmnet
  • 13:35 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir1002.eqiad.wmnet with OS trixie
  • 13:30 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs-test1001.eqiad.wmnet
  • 13:29 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs-test1001.eqiad.wmnet
  • 13:29 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P{dse-k8s-wdqs-test1001.eqiad.wmnet} and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad)
  • 13:25 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from codfw to eqiad for section test-s4
  • 13:24 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from codfw to eqiad for section test-s4
  • 13:24 jelto@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on wikikube-worker1152.eqiad.wmnet with reason: hardware/networking issues
  • 13:18 sukhe@cumin1004: START - Cookbook sre.hosts.reimage for host sretest2013.codfw.wmnet with OS trixie
  • 13:13 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from codfw to eqiad for section test-s4
  • 13:10 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from codfw to eqiad for section test-s4
  • 13:09 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from eqiad to codfw for section test-s4
  • 13:04 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.finalize for the switch from eqiad to codfw for section test-s4
  • 12:57 brouberol@cumin1004: END (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99)
  • 12:57 brouberol@cumin1004: START - Cookbook sre.ceph.remove-osd
  • 12:56 awight: manually run puppet agent
  • 12:56 brouberol@cumin1004: END (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99)
  • 12:56 brouberol@cumin1004: START - Cookbook sre.ceph.remove-osd
  • 12:55 brouberol@cumin1004: END (PASS) - Cookbook sre.ceph.remove-osd (exit_code=0)
  • 12:55 brouberol@cumin1004: START - Cookbook sre.ceph.remove-osd
  • 12:45 awight: add seanleong-wmde to deployment-prep
  • 12:44 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr1-eqiad,ssw1-d[1,8]-eqiad with reason: re-rack ssw1-a1-eqiad
  • 12:39 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section test-s4
  • 12:39 brouberol@cumin1004: END (PASS) - Cookbook sre.ceph.remove-osd (exit_code=0)
  • 12:38 brouberol@cumin1004: START - Cookbook sre.ceph.remove-osd
  • 12:34 kharlan@deploy1003: Finished scap sync-world: Backport for AbuseReview: Add warning indicating alpha test to vandalism queue (T438467) (duration: 33m 33s)
  • 12:33 brouberol@cumin1004: END (PASS) - Cookbook sre.ceph.remove-osd (exit_code=0)
  • 12:33 brouberol@cumin1004: START - Cookbook sre.ceph.remove-osd
  • 12:32 brouberol@cumin1004: END (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99)
  • 12:32 brouberol@cumin1004: START - Cookbook sre.ceph.remove-osd
  • 12:32 brouberol@cumin1004: END (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99)
  • 12:32 brouberol@cumin1004: START - Cookbook sre.ceph.remove-osd
  • 12:30 brouberol@cumin1004: END (PASS) - Cookbook sre.ceph.remove-osd (exit_code=0)
  • 12:30 brouberol@cumin1004: START - Cookbook sre.ceph.remove-osd
  • 12:29 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section test-s4
  • 12:22 kharlan@deploy1003: kharlan: Continuing with deployment
  • 12:21 kharlan@deploy1003: kharlan: Backport for AbuseReview: Add warning indicating alpha test to vandalism queue (T438467) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 12:15 cdanis@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool eqiad [reason: no reason specified, no task ID specified]
  • 12:15 cdanis@cumin1004: START - Cookbook sre.dns.admin DNS admin: pool eqiad [reason: no reason specified, no task ID specified]
  • 12:01 kharlan@deploy1003: Started scap sync-world: Backport for AbuseReview: Add warning indicating alpha test to vandalism queue (T438467)
  • 11:51 Dreamy_Jazz: Deployed patch for T438729
  • 11:31 mvolz@deploy1003: helmfile [eqiad] DONE helmfile.d/services/citoid: apply
  • 11:28 mvolz@deploy1003: helmfile [eqiad] START helmfile.d/services/citoid: apply
  • 11:27 mvolz@deploy1003: helmfile [codfw] DONE helmfile.d/services/citoid: apply
  • 11:27 mvolz@deploy1003: helmfile [codfw] START helmfile.d/services/citoid: apply
  • 11:25 mvolz@deploy1003: helmfile [staging] DONE helmfile.d/services/citoid: apply
  • 11:25 mvolz@deploy1003: helmfile [staging] START helmfile.d/services/citoid: apply
  • 10:38 jayme: sudo confctl --quiet --object-type discovery select 'dnsdisc=mw-web-ro' set/ttl=10 - T438896
  • 10:31 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative: apply
  • 10:31 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative: apply
  • 10:25 blake@deploy1003: Finished scap sync-world: Upsize mw-web T438896 (duration: 04m 20s)
  • 10:22 blake@deploy1003: Started scap sync-world: Upsize mw-web T438896
  • 10:06 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P{dse-k8s-wdqs100[1-3].eqiad.wmnet} and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad)
  • 10:06 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1003.eqiad.wmnet
  • 10:06 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1003.eqiad.wmnet
  • 10:04 ayounsi@cumin1004: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Add netbox-bgp and update wheelson netbox-next - ayounsi@cumin1004
  • 09:59 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1003.eqiad.wmnet
  • 09:59 ayounsi@cumin1004: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Add netbox-bgp and update wheelson netbox-next - ayounsi@cumin1004
  • 09:58 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1003.eqiad.wmnet
  • 09:58 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1002.eqiad.wmnet
  • 09:58 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1002.eqiad.wmnet
  • 09:57 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99)
  • 09:55 brouberol@cumin1004: DONE (PASS) - Cookbook sre.ceph.remove-osd (exit_code=0)
  • 09:54 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99)
  • 09:54 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99)
  • 09:53 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99)
  • 09:52 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99)
  • 09:51 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99)
  • 09:51 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1002.eqiad.wmnet
  • 09:51 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1002.eqiad.wmnet
  • 09:51 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1001.eqiad.wmnet
  • 09:51 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1001.eqiad.wmnet
  • 09:50 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99)
  • 09:44 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1001.eqiad.wmnet
  • 09:43 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1001.eqiad.wmnet
  • 09:43 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P{dse-k8s-wdqs100[1-3].eqiad.wmnet} and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad)
  • 09:38 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99)
  • 09:34 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99)
  • 08:45 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99)
  • 08:44 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99)
  • 08:44 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99)
  • 08:41 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99)
  • 08:27 brouberol@cumin1004: END (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99)
  • 08:27 brouberol@cumin1004: START - Cookbook sre.ceph.remove-osd
  • 08:25 kevinbazira@deploy1003: helmfile [ml-serve-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' .
  • 08:24 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'tts-section-generator' for release 'main' .
  • 08:13 tappof@deploy1003: Finished scap sync-world: T432444 - Provision kafka-logging100[6-8] (duration: 12m 52s)
  • 08:05 moritzm: installing grub2 bugfix updates on Bookworm hosts
  • 08:04 tappof@deploy1003: Started scap sync-world: T432444 - Provision kafka-logging100[6-8]
  • 08:00 tappof@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'sync'.
  • 07:59 tappof@deploy1003: helmfile [codfw] START helmfile.d/admin 'sync'.
  • 07:59 moritzm: installing giflib security updates
  • 07:58 tappof@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'sync'.
  • 07:58 tappof@deploy1003: helmfile [eqiad] START helmfile.d/admin 'sync'.
  • 07:29 moritzm: installing python-idna security updates
  • 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 39s)
  • 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image
  • 00:50 ryankemper@cumin2003: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool wdqs-main in eqiad: maintenance
  • 00:46 ryankemper: [WDQS] T435443 Restore eqiad wdqs-main; wdqs was unable to keep up with traffic with only one datacenter. sadly this will continue to be the case until wdqsv2 is ready to switch backend architecture
  • 00:45 ryankemper@cumin2003: START - Cookbook sre.discovery.service-route pool wdqs-main in eqiad: maintenance

2026-09-22

  • 23:23 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P{dse-k8s-worker10[02-28].eqiad.wmnet} and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad)
  • 23:23 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1028.eqiad.wmnet
  • 23:23 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1028.eqiad.wmnet
  • 23:15 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1028.eqiad.wmnet
  • 22:45 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1028.eqiad.wmnet
  • 22:45 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1027.eqiad.wmnet
  • 22:45 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1027.eqiad.wmnet
  • 22:36 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1027.eqiad.wmnet
  • 22:30 ryankemper: [WDQS] codfw wdqs-main is struggling under the switchover load, fiddling with some auto-restart knobs to see if it helps or hurts
  • 22:06 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1027.eqiad.wmnet
  • 22:06 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1026.eqiad.wmnet
  • 22:06 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1026.eqiad.wmnet
  • 21:58 rzl@deploy1003: Finished scap sync-world: https://gerrit.wikimedia.org/r/1339694 T437403 (duration: 13m 43s)
  • 21:57 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1026.eqiad.wmnet
  • 21:57 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1026.eqiad.wmnet
  • 21:57 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1025.eqiad.wmnet
  • 21:57 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1025.eqiad.wmnet
  • 21:53 rzl@deploy1003: rzl: Continuing with deployment
  • 21:51 rzl@deploy1003: rzl: https://gerrit.wikimedia.org/r/1339694 T437403 synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 21:49 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1025.eqiad.wmnet
  • 21:47 rzl@deploy1003: Started scap sync-world: https://gerrit.wikimedia.org/r/1339694 T437403
  • 21:25 aqu@deploy1003: Finished deploy [analytics/refinery@58c9356] (thin): Regular analytics weekly train THIN [analytics/refinery@58c93566] (duration: 01m 09s)
  • 21:24 aqu@deploy1003: Started deploy [analytics/refinery@58c9356] (thin): Regular analytics weekly train THIN [analytics/refinery@58c93566]
  • 21:24 aqu@deploy1003: Finished deploy [analytics/refinery@58c9356] (thin): Regular analytics weekly train THIN [analytics/refinery@58c93566] (duration: 24m 20s)
  • 21:19 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1025.eqiad.wmnet
  • 21:18 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1024.eqiad.wmnet
  • 21:18 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1024.eqiad.wmnet
  • 21:10 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1024.eqiad.wmnet
  • 21:05 sukhe@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2013.codfw.wmnet with OS trixie
  • 20:59 aqu@deploy1003: Started deploy [analytics/refinery@58c9356] (thin): Regular analytics weekly train THIN [analytics/refinery@58c93566]
  • 20:59 aqu@deploy1003: Finished deploy [analytics/refinery@58c9356] (thin): Regular analytics weekly train THIN [analytics/refinery@58c93566] (duration: 00m 30s)
  • 20:59 aqu@deploy1003: Started deploy [analytics/refinery@58c9356] (thin): Regular analytics weekly train THIN [analytics/refinery@58c93566]
  • 20:55 aqu@deploy1003: Finished deploy [analytics/refinery@58c9356]: Regular analytics weekly train [analytics/refinery@58c93566] (duration: 06m 59s)
  • 20:48 aqu@deploy1003: Started deploy [analytics/refinery@58c9356]: Regular analytics weekly train [analytics/refinery@58c93566]
  • 20:46 aqu@deploy1003: Finished deploy [analytics/refinery@58c9356] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@58c93566] (duration: 00m 40s)
  • 20:45 bking@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'.
  • 20:45 sbisson@deploy1003: Finished scap sync-world: Backport for Keep Article Guidance on where it is on today (T433293) (duration: 09m 53s)
  • 20:45 aqu@deploy1003: Started deploy [analytics/refinery@58c9356] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@58c93566]
  • 20:44 bking@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'.
  • 20:44 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
  • 20:43 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
  • 20:40 sbisson@deploy1003: sbisson: Continuing with deployment
  • 20:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1024.eqiad.wmnet
  • 20:40 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1023.eqiad.wmnet
  • 20:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1023.eqiad.wmnet
  • 20:40 sbisson@deploy1003: sbisson: Backport for Keep Article Guidance on where it is on today (T433293) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 20:35 sbisson@deploy1003: Started scap sync-world: Backport for Keep Article Guidance on where it is on today (T433293)
  • 20:33 ebernhardson@deploy1003: Finished scap sync-world: Backport for eswiki: Add abusefilter-access-protected-vars to abusefilter user group (T436652) (duration: 13m 35s)
  • 20:33 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1023.eqiad.wmnet
  • 20:28 ebernhardson@deploy1003: ebernhardson, codenamenoreste: Continuing with deployment
  • 20:24 ebernhardson@deploy1003: ebernhardson, codenamenoreste: Backport for eswiki: Add abusefilter-access-protected-vars to abusefilter user group (T436652) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 20:20 ebernhardson@deploy1003: Started scap sync-world: Backport for eswiki: Add abusefilter-access-protected-vars to abusefilter user group (T436652)
  • 20:17 ebernhardson@deploy1003: Finished scap sync-world: Backport for cirrus: Send more_like traffic to eqiad (duration: 10m 29s)
  • 20:15 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
  • 20:13 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir2002.*
  • 20:12 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
  • 20:12 ebernhardson@deploy1003: ebernhardson: Continuing with deployment
  • 20:11 ebernhardson@deploy1003: ebernhardson: Backport for cirrus: Send more_like traffic to eqiad synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 20:06 ebernhardson@deploy1003: Started scap sync-world: Backport for cirrus: Send more_like traffic to eqiad
  • 20:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1023.eqiad.wmnet
  • 20:02 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1022.eqiad.wmnet
  • 20:02 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1022.eqiad.wmnet
  • 20:02 sukhe@cumin1004: START - Cookbook sre.hosts.reimage for host sretest2013.codfw.wmnet with OS trixie
  • 19:59 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir2002.codfw.wmnet with OS trixie
  • 19:44 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1022.eqiad.wmnet
  • 19:42 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir2002.codfw.wmnet with reason: host reimage
  • 19:42 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1022.eqiad.wmnet
  • 19:42 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1021.eqiad.wmnet
  • 19:42 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1021.eqiad.wmnet
  • 19:38 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir2002.codfw.wmnet with reason: host reimage
  • 19:34 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-serve1016.eqiad.wmnet with OS trixie
  • 19:34 jclark@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1004"
  • 19:33 jclark@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1004"
  • 19:25 sukhe@cumin1004: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from sretest2013 to cp2059
  • 19:24 sukhe@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059
  • 19:23 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1021.eqiad.wmnet
  • 19:22 sukhe@cumin1004: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from sretest2013 to cp2059
  • 19:21 sukhe@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059
  • 19:19 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir2002.codfw.wmnet with OS trixie
  • 19:19 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-serve1016.eqiad.wmnet with reason: host reimage
  • 19:17 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1021.eqiad.wmnet
  • 19:17 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1020.eqiad.wmnet
  • 19:17 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1020.eqiad.wmnet
  • 19:15 jclark@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ml-serve1016.eqiad.wmnet with reason: host reimage
  • 19:01 ebernhardson: Rolling restart opensearch-semantic-search in dse-k8s-codfw to update to opensearch 3.8.0
  • 18:58 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1020.eqiad.wmnet
  • 18:56 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1020.eqiad.wmnet
  • 18:56 jclark@cumin1004: START - Cookbook sre.hosts.reimage for host ml-serve1016.eqiad.wmnet with OS trixie
  • 18:56 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1019.eqiad.wmnet
  • 18:56 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1019.eqiad.wmnet
  • 18:55 dancy@deploy1003: Installation of scap version "4.291.0" completed for 2 hosts
  • 18:53 dancy@deploy1003: Installing scap version "4.291.0" for 2 host(s)
  • 18:53 dancy@deploy1003: Installation of scap version "4.291.0" completed for 3 hosts
  • 18:51 dancy@deploy1003: Installing scap version "4.291.0" for 3 host(s)
  • 18:49 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1019.eqiad.wmnet
  • 18:49 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1019.eqiad.wmnet
  • 18:49 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1018.eqiad.wmnet
  • 18:49 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1018.eqiad.wmnet
  • 18:47 dancy@deploy1003: Installing scap version "4.291.0" for 3 host(s)
  • 18:44 dancy@deploy1003: Installing scap version "4.291.0" for 3 host(s)
  • 18:43 dancy@deploy1003: Installing scap version "4.291.0" for 3 host(s)
  • 18:41 dancy@deploy1003: install-world aborted: (no justification provided) (duration: 00m 48s)
  • 18:41 dancy@deploy1003: Installing scap version "4.291.0" for 3 host(s)
  • 18:40 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1018.eqiad.wmnet
  • 18:36 jhuneidi@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.47.0-wmf.21 refs T438217
  • 18:35 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1018.eqiad.wmnet
  • 18:35 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1014.eqiad.wmnet
  • 18:35 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1014.eqiad.wmnet
  • 18:18 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1014.eqiad.wmnet
  • 18:16 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1014.eqiad.wmnet
  • 18:16 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1013.eqiad.wmnet
  • 18:16 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1013.eqiad.wmnet
  • 18:09 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1013.eqiad.wmnet
  • 18:07 ebernhardson: Rolling restart opensearch-semantic-search in dse-k8s-eqiad to update to opensearch 3.8.0
  • 17:55 dreamyjazz@deploy1003: Finished scap sync-world: Backport for fix(WikimediaAntiAbuse): use correct endpoint for LiftWing in eqiad (duration: 10m 09s)
  • 17:54 bking@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wdqs1030
  • 17:54 bking@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wdqs1030
  • 17:53 bking@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host wdqs1030
  • 17:53 bking@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wdqs1030.eqiad.wmnet 8.32.64.10.in-addr.arpa 8.0.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
  • 17:53 bking@cumin2003: START - Cookbook sre.dns.wipe-cache wdqs1030.eqiad.wmnet 8.32.64.10.in-addr.arpa 8.0.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
  • 17:53 bking@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 17:53 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wdqs1030 - bking@cumin2003"
  • 17:53 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wdqs1030 - bking@cumin2003"
  • 17:51 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2218: Optimizer issues fixed
  • 17:50 dreamyjazz@deploy1003: dreamyjazz, isaranto: Continuing with deployment
  • 17:50 dreamyjazz@deploy1003: dreamyjazz, isaranto: Backport for fix(WikimediaAntiAbuse): use correct endpoint for LiftWing in eqiad synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 17:47 sukhe@cumin1004: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from sretest2013 to cp2059
  • 17:46 sukhe@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059
  • 17:45 dreamyjazz@deploy1003: Started scap sync-world: Backport for fix(WikimediaAntiAbuse): use correct endpoint for LiftWing in eqiad
  • 17:45 bking@cumin2003: START - Cookbook sre.dns.netbox
  • 17:43 bking@cumin2003: START - Cookbook sre.hosts.move-vlan for host wdqs1030
  • 17:39 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1013.eqiad.wmnet
  • 17:39 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1012.eqiad.wmnet
  • 17:39 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1012.eqiad.wmnet
  • 17:37 bking@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wdqs1029
  • 17:37 bking@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wdqs1029
  • 17:36 bking@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host wdqs1029
  • 17:36 bking@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wdqs1029.eqiad.wmnet 8.48.64.10.in-addr.arpa 8.0.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
  • 17:36 bking@cumin2003: START - Cookbook sre.dns.wipe-cache wdqs1029.eqiad.wmnet 8.48.64.10.in-addr.arpa 8.0.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
  • 17:36 bking@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 17:36 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wdqs1029 - bking@cumin2003"
  • 17:36 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wdqs1029 - bking@cumin2003"
  • 17:33 sukhe@cumin1004: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from sretest2013 to cp2059
  • 17:32 sukhe@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059
  • 17:31 bking@cumin2003: START - Cookbook sre.dns.netbox
  • 17:31 bking@cumin2003: START - Cookbook sre.hosts.move-vlan for host wdqs1029
  • 17:26 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1012.eqiad.wmnet
  • 17:25 sukhe@cumin1004: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from sretest2013 to cp2059
  • 17:25 sukhe@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059
  • 17:24 dzahn@dns1004: END - running authdns-update
  • 17:24 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1012.eqiad.wmnet
  • 17:24 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1011.eqiad.wmnet
  • 17:24 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1011.eqiad.wmnet
  • 17:22 dzahn@dns1004: START - running authdns-update
  • 17:17 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1011.eqiad.wmnet
  • 17:17 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1011.eqiad.wmnet
  • 17:16 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1010.eqiad.wmnet
  • 17:16 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1010.eqiad.wmnet
  • 17:15 oblivian@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=rest-gateway,name=codfw
  • 17:10 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1010.eqiad.wmnet
  • 17:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1010.eqiad.wmnet
  • 17:09 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1009.eqiad.wmnet
  • 17:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1009.eqiad.wmnet
  • 17:06 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db2218: Optimizer issues fixed
  • 17:03 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1009.eqiad.wmnet
  • 17:02 sukhe@cumin1004: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from sretest2013 to cp2059.codfw.wmnet
  • 17:01 sukhe@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059.codfw.wmnet
  • 17:01 sukhe@cumin1004: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from sretest2013 to cp2059
  • 17:00 sukhe@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059
  • 16:59 dreamyjazz@deploy1003: Finished scap sync-world: Backport for Enable AbuseReview on jawiki for likely PII (T438867) (duration: 13m 13s)
  • 16:54 oblivian@cumin1004: END (FAIL) - Cookbook sre.discovery.service-route (exit_code=99) pool 2 services in eqiad: maintenance
  • 16:51 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment
  • 16:50 dreamyjazz@deploy1003: dreamyjazz: Backport for Enable AbuseReview on jawiki for likely PII (T438867) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 16:48 oblivian@cumin1004: START - Cookbook sre.discovery.service-route pool 2 services in eqiad: maintenance
  • 16:46 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2218.codfw.wmnet with reason: fixing
  • 16:45 dreamyjazz@deploy1003: Started scap sync-world: Backport for Enable AbuseReview on jawiki for likely PII (T438867)
  • 16:42 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2218.codfw.wmnet with reason: fixing
  • 16:42 cdobbins@cumin1004: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from sretest2013 to cp2059
  • 16:41 cdobbins@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059
  • 16:33 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1009.eqiad.wmnet
  • 16:33 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1008.eqiad.wmnet
  • 16:33 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1008.eqiad.wmnet
  • 16:26 cdobbins@cumin1004: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from sretest2013 to cp2059
  • 16:26 cdobbins@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059
  • 16:25 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1008.eqiad.wmnet
  • 16:19 oblivian@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool 4 services in eqiad: maintenance
  • 16:13 oblivian@cumin1004: START - Cookbook sre.discovery.service-route pool 4 services in eqiad: maintenance
  • 16:04 sukhe@cumin1004: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from sretest2013 to cp2059
  • 16:04 sukhe@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059
  • 15:58 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2218: optimizer issues
  • 15:57 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool db2218: optimizer issues
  • 15:55 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1008.eqiad.wmnet
  • 15:55 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1007.eqiad.wmnet
  • 15:55 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1007.eqiad.wmnet
  • 15:50 moritzm: installing libhtml-parser-perl security updates
  • 15:49 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1007.eqiad.wmnet
  • 15:40 oblivian@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool mw-web-ro in eqiad: maintenance
  • 15:36 ayounsi@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 15:36 ayounsi@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cirrussearch1120 move vlan - ayounsi@cumin1004"
  • 15:36 ayounsi@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cirrussearch1120 move vlan - ayounsi@cumin1004"
  • 15:35 oblivian@cumin1004: START - Cookbook sre.discovery.service-route pool mw-web-ro in eqiad: maintenance
  • 15:35 oblivian@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) check mw-web-ro: maintenance
  • 15:35 oblivian@cumin1004: START - Cookbook sre.discovery.service-route check mw-web-ro: maintenance
  • 15:27 ayounsi@cumin1004: START - Cookbook sre.dns.netbox
  • 15:22 slyngshede@cumin1004: END (PASS) - Cookbook sre.discovery.datacenter (exit_code=0) depool all services in eqiad: Datacenter services switchover - T435443
  • 15:19 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1007.eqiad.wmnet
  • 15:18 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1006.eqiad.wmnet
  • 15:18 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1006.eqiad.wmnet
  • 15:16 bking@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cirrussearch1120
  • 15:16 bking@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host cirrussearch1120
  • 15:14 bking@cumin2003: END (FAIL) - Cookbook sre.hosts.move-vlan (exit_code=99) for host cirrussearch1120
  • 15:11 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1006.eqiad.wmnet
  • 15:11 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1006.eqiad.wmnet
  • 15:11 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1005.eqiad.wmnet
  • 15:11 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1005.eqiad.wmnet
  • 15:04 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1005.eqiad.wmnet
  • 15:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1005.eqiad.wmnet
  • 15:03 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1004.eqiad.wmnet
  • 15:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1004.eqiad.wmnet
  • 15:01 dancy@deploy1003: Installation of scap version "4.290.0" completed for 3 hosts
  • 14:59 dancy@deploy1003: Installing scap version "4.290.0" for 3 host(s)
  • 14:55 slyngshede@cumin1004: START - Cookbook sre.discovery.datacenter depool all services in eqiad: Datacenter services switchover - T435443
  • 14:55 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1004.eqiad.wmnet
  • 14:54 dancy@deploy1003: Installing scap version "4.290.0" for 155 host(s)
  • 14:54 slyngshede@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool eqiad [reason: no reason specified, no task ID specified]
  • 14:54 slyngshede@cumin1004: START - Cookbook sre.dns.admin DNS admin: depool eqiad [reason: no reason specified, no task ID specified]
  • 14:53 bking@cumin2003: START - Cookbook sre.hosts.move-vlan for host cirrussearch1120
  • 14:51 bking@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch1120.eqiad.wmnet with reason: migrate VLAN T436571
  • 14:47 bking@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cirrussearch1120
  • 14:47 bking@cumin2003: START - Cookbook sre.hosts.move-vlan for host cirrussearch1120
  • 14:42 cdobbins@cumin1004: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from sretest2013 to cp2059
  • 14:42 cdobbins@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059
  • 14:36 cdobbins@cumin1004: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from sretest2013 to cp2059
  • 14:35 cdobbins@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059
  • 14:25 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1004.eqiad.wmnet
  • 14:25 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1003.eqiad.wmnet
  • 14:25 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1003.eqiad.wmnet
  • 14:17 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1003.eqiad.wmnet
  • 14:15 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1003.eqiad.wmnet
  • 14:15 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1002.eqiad.wmnet
  • 14:15 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1002.eqiad.wmnet
  • 13:59 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1002.eqiad.wmnet
  • 13:57 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1002.eqiad.wmnet
  • 13:57 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P{dse-k8s-worker10[02-28].eqiad.wmnet} and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad)
  • 13:57 tappof@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'sync'.
  • 13:56 tappof@deploy1003: helmfile [codfw] START helmfile.d/admin 'sync'.
  • 13:56 tappof@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'sync'.
  • 13:55 tappof@deploy1003: helmfile [eqiad] START helmfile.d/admin 'sync'.
  • 13:53 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.powercycle (exit_code=0) for host pki1002
  • 13:51 elukey@cumin1004: START - Cookbook sre.hosts.powercycle for host pki1002
  • 13:23 tappof@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'sync'.
  • 13:22 tappof@deploy1003: helmfile [codfw] START helmfile.d/admin 'sync'.
  • 13:21 tappof@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'sync'.
  • 13:21 tappof@deploy1003: helmfile [eqiad] START helmfile.d/admin 'sync'.
  • 12:53 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-ctrl1001.eqiad.wmnet
  • 12:48 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-ctrl1001.eqiad.wmnet
  • 12:44 marostegui: Stop mariadb on db2250:s5 T437411 T437279
  • 12:43 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2250.codfw.wmnet with reason: preparations
  • 12:31 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P{dse-k8s-worker1001.eqiad.wmnet} and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad)
  • 12:31 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1001.eqiad.wmnet
  • 12:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1001.eqiad.wmnet
  • 12:22 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1001.eqiad.wmnet
  • 12:19 kharlan@deploy1003: Finished scap sync-world: Backport for AbuseReview: Hide recently saved revisions from the vandalism queue (T438235) (duration: 33m 01s)
  • 12:17 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'.
  • 12:16 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'.
  • 12:16 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
  • 12:15 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
  • 12:08 kharlan@deploy1003: kharlan: Continuing with deployment
  • 12:06 kharlan@deploy1003: kharlan: Backport for AbuseReview: Hide recently saved revisions from the vandalism queue (T438235) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 11:54 jnuche@deploy1003: Finished deploy [releng/jenkins-deploy@ddb3f1a] (releasing): T435791 to production host (duration: 00m 54s)
  • 11:54 jnuche@deploy1003: Started deploy [releng/jenkins-deploy@ddb3f1a] (releasing): T435791 to production host
  • 11:52 jnuche@deploy1003: Finished deploy [releng/jenkins-deploy@ddb3f1a] (releasing): T435791 to backup host (duration: 01m 01s)
  • 11:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1001.eqiad.wmnet
  • 11:52 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P{dse-k8s-worker1001.eqiad.wmnet} and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad)
  • 11:52 jnuche@deploy1003: Started deploy [releng/jenkins-deploy@ddb3f1a] (releasing): T435791 to backup host
  • 11:46 kharlan@deploy1003: Started scap sync-world: Backport for AbuseReview: Hide recently saved revisions from the vandalism queue (T438235)
  • 11:41 kharlan@deploy1003: Finished scap sync-world: Backport for AbuseReview: Hide Echo banner when user cannot see personal info (T438477) (duration: 13m 46s)
  • 11:34 kharlan@deploy1003: kharlan: Continuing with deployment
  • 11:33 kharlan@deploy1003: kharlan: Backport for AbuseReview: Hide Echo banner when user cannot see personal info (T438477) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 11:27 kharlan@deploy1003: Started scap sync-world: Backport for AbuseReview: Hide Echo banner when user cannot see personal info (T438477)
  • 11:24 kharlan@deploy1003: Finished scap sync-world: Backport for AbuseReview: Allow interaction with verdict buttons on closed rows (T438808) (duration: 33m 09s)
  • 11:24 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-eqiad@eqiad
  • 11:24 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs
  • 11:22 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs
  • 11:19 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad
  • 11:13 kharlan@deploy1003: kharlan: Continuing with deployment
  • 11:12 kharlan@deploy1003: kharlan: Backport for AbuseReview: Allow interaction with verdict buttons on closed rows (T438808) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 10:54 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
  • 10:54 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
  • 10:53 gmodena@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply
  • 10:53 gmodena@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply
  • 10:52 topranks: enable rule cache-upload/eqsin_originals_scraper_20260922
  • 10:51 kharlan@deploy1003: Started scap sync-world: Backport for AbuseReview: Allow interaction with verdict buttons on closed rows (T438808)
  • 10:20 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host registry2005.codfw.wmnet with OS trixie
  • 10:13 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section s1
  • 10:11 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section s1
  • 10:10 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section s4
  • 10:10 gmodena@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply
  • 10:09 gmodena@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply
  • 10:09 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section s4
  • 10:09 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
  • 10:09 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
  • 10:08 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section s8
  • 10:06 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section s8
  • 10:06 moritzm: installing libcap2 security updates
  • 10:05 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section s7
  • 10:03 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section s7
  • 10:02 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on registry2005.codfw.wmnet with reason: host reimage
  • 10:02 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section s3
  • 10:01 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section s3
  • 10:00 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section s2
  • 09:58 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section s2
  • 09:58 elukey@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on registry2005.codfw.wmnet with reason: host reimage
  • 09:57 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section s5
  • 09:56 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on P{cp[7010,7016].*} and A:cp - 3.2.23 upgrade (T438828)
  • 09:55 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section s5
  • 09:53 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section s6
  • 09:51 elukey: install spicerack 13.3.0 on cumin1004 and cumin2003
  • 09:50 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section s6
  • 09:47 elukey: uploaded spicerack_13.3.0 to apt.wikimedia.org bookworm-wikimedia,trixie-wikimedia
  • 09:47 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1035: issues
  • 09:46 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool es1035: issues
  • 09:44 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section es7
  • 09:44 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on P{cp[7010,7016].*} and A:cp - 3.2.23 upgrade (T438828)
  • 09:44 kevinbazira@deploy1003: helmfile [ml-serve-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' .
  • 09:43 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'tts-section-generator' for release 'main' .
  • 09:42 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section es7
  • 09:41 kevinbazira@deploy1003: helmfile [ml-staging-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' .
  • 09:41 elukey@cumin1004: START - Cookbook sre.hosts.reimage for host registry2005.codfw.wmnet with OS trixie
  • 09:40 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section es7
  • 09:39 vgutierrez: fetch haproxy 3.2.23 on thirdparty/haproxy32 for trixie (apt.wm.o) - T438828
  • 09:32 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es1035: issues
  • 09:32 jelto@cumin1004: END (FAIL) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=99) for alias: wikikube-worker-eqiad@eqiad
  • 09:32 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool es1035: issues
  • 09:31 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 8 hosts with reason: dc preparations
  • 09:30 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad
  • 09:28 jelto@cumin1004: conftool action : set/pooled=inactive; selector: name=wikikube-worker1152.eqiad.wmnet
  • 09:28 jelto@cumin1004: END (FAIL) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=99) for alias: wikikube-worker-eqiad@eqiad
  • 09:26 btullis@dns1004: END - running authdns-update
  • 09:24 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad
  • 09:23 btullis@dns1004: START - running authdns-update
  • 09:23 jelto@cumin1004: conftool action : set/pooled=no; selector: name=wikikube-worker1152.eqiad.wmnet
  • 09:20 jelto@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on wikikube-worker1152.eqiad.wmnet with reason: hardware/networking issues
  • 09:16 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section es7
  • 09:15 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section es6
  • 09:14 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section es6
  • 09:12 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section x4
  • 09:11 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section x4
  • 09:11 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section x3
  • 09:10 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 52320
  • 09:09 ayounsi@cumin1004: START - Cookbook sre.network.peering with action 'email' for AS: 52320
  • 09:05 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section x3
  • 09:04 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section x1
  • 09:02 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section x1
  • 08:58 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply
  • 08:55 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply
  • 08:52 trueg@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply
  • 08:49 trueg@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply
  • 08:45 ayounsi@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool esams [reason: switch reboot, T437984]
  • 08:45 ayounsi@cumin1004: START - Cookbook sre.dns.admin DNS admin: pool esams [reason: switch reboot, T437984]
  • 08:44 ayounsi@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for asw1-bw27-esams,asw1-bw27-esams IPv6,asw1-bw27-esams.mgmt
  • 08:44 ayounsi@cumin1004: START - Cookbook sre.hosts.remove-downtime for asw1-bw27-esams,asw1-bw27-esams IPv6,asw1-bw27-esams.mgmt
  • 08:44 ayounsi@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for 13 hosts
  • 08:44 ayounsi@cumin1004: START - Cookbook sre.hosts.remove-downtime for 13 hosts
  • 08:39 trueg@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply
  • 08:39 trueg@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply
  • 08:37 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
  • 08:37 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
  • 08:32 moritzm: installig zip security updates
  • 08:30 jelto@cumin1004: END (FAIL) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=99) for alias: wikikube-worker-eqiad@eqiad
  • 08:29 XioNoX: asw1-bw27-esams> request system reboot - T437984
  • 08:28 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad
  • 08:27 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.depool-rack (exit_code=0) with action 'depool' for esams rack BW27
  • 08:26 jelto@cumin1004: END (FAIL) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=99) for alias: wikikube-worker-eqiad@eqiad
  • 08:26 ayounsi@cumin1004: START - Cookbook sre.network.depool-rack with action 'depool' for esams rack BW27
  • 08:24 dcausse@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply
  • 08:24 dcausse@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply
  • 08:22 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad
  • 08:18 moritzm: installing gst-plugins-base1.0 security updates
  • 08:10 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-codfw@codfw
  • 08:10 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs
  • 08:09 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs
  • 08:05 arthurtaylor@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply
  • 08:05 arthurtaylor@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply
  • 08:05 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-codfw@codfw
  • 08:04 arthurtaylor@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikidata-query-gui: apply
  • 08:04 arthurtaylor@deploy1003: helmfile [codfw] START helmfile.d/services/wikidata-query-gui: apply
  • 08:01 arthurtaylor@deploy1003: helmfile [staging] DONE helmfile.d/services/wikidata-query-gui: apply
  • 08:01 ayounsi@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 13 hosts with reason: Switch reboot
  • 08:01 arthurtaylor@deploy1003: helmfile [staging] START helmfile.d/services/wikidata-query-gui: apply
  • 08:01 ayounsi@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on asw1-bw27-esams,asw1-bw27-esams IPv6,asw1-bw27-esams.mgmt with reason: Switch reboot
  • 07:59 ayounsi@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool esams [reason: switch reboot, T437984]
  • 07:59 ayounsi@cumin1004: START - Cookbook sre.dns.admin DNS admin: depool esams [reason: switch reboot, T437984]
  • 07:23 awight: UTC morning deployment window complete
  • 07:22 awight@deploy1003: Finished scap sync-world: Backport for Config change for launch of stopping sending LL notifications. (T438463), Change feedback URLs for EditCheck TextMatch on ruwiki (T426271) (duration: 17m 46s)
  • 07:15 awight@deploy1003: seanleong-wmde, esanders, awight: Continuing with deployment
  • 07:09 awight@deploy1003: seanleong-wmde, esanders, awight: Backport for Config change for launch of stopping sending LL notifications. (T438463), Change feedback URLs for EditCheck TextMatch on ruwiki (T426271) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 07:05 awight@deploy1003: Started scap sync-world: Backport for Config change for launch of stopping sending LL notifications. (T438463), Change feedback URLs for EditCheck TextMatch on ruwiki (T426271)
  • 07:02 moritzm: installing pyasn1 security updates
  • 04:02 mwpresync@deploy1003: Pruned MediaWiki: 1.47.0-wmf.18 (duration: 02m 28s)
  • 03:39 mwpresync@deploy1003: Finished scap sync-world: testwikis to 1.47.0-wmf.21 refs T438217 (duration: 35m 52s)
  • 03:03 mwpresync@deploy1003: Started scap sync-world: testwikis to 1.47.0-wmf.21 refs T438217
  • 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 30s)
  • 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image

2026-09-21

  • 22:11 rzl@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
  • 22:10 rzl@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'.
  • 22:09 rzl@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'.
  • 22:08 rzl@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'.
  • 22:08 rzl@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'.
  • 22:07 rzl@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'.
  • 22:06 rzl@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
  • 22:05 rzl@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
  • 21:18 maryum: Deployed security fix for T437708
  • 20:35 cjming@deploy1003: Finished scap sync-world: Backport for Disable wgMFCustomSiteModules on German Wikipedia (T403380) (duration: 15m 56s)
  • 20:30 cjming@deploy1003: ameisenigel, cjming: Continuing with deployment
  • 20:26 ihurbain@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
  • 20:25 ihurbain@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
  • 20:25 ihurbain@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
  • 20:25 ihurbain@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
  • 20:23 cjming@deploy1003: ameisenigel, cjming: Backport for Disable wgMFCustomSiteModules on German Wikipedia (T403380) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 20:19 cjming@deploy1003: Started scap sync-world: Backport for Disable wgMFCustomSiteModules on German Wikipedia (T403380)
  • 19:02 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply
  • 19:02 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply
  • 18:59 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply
  • 18:59 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply
  • 18:35 mvernon@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool sessionstore in eqiad: sessionstore1005 repaired
  • 18:32 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir5003.*
  • 18:30 Emperor: repool eqiad sessionstore T437915
  • 18:30 mvernon@cumin1004: START - Cookbook sre.discovery.service-route pool sessionstore in eqiad: sessionstore1005 repaired
  • 18:27 mvernon@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) check sessionstore: maintenance
  • 18:27 mvernon@cumin1004: START - Cookbook sre.discovery.service-route check sessionstore: maintenance
  • 18:25 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply
  • 18:25 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply
  • 18:24 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir5003.eqsin.wmnet with OS trixie
  • 17:54 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir5003.eqsin.wmnet with reason: host reimage
  • 17:50 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir5003.eqsin.wmnet with reason: host reimage
  • 17:40 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sessionstore1005.eqiad.wmnet with OS bookworm
  • 17:30 jclark@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sessionstore1005.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 17:29 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
  • 17:26 jclark@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
  • 17:12 jclark@cumin1004: START - Cookbook sre.hosts.provision for host sessionstore1005.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 17:00 jclark@cumin1004: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bookworm
  • 16:56 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply
  • 16:56 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply
  • 16:54 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir5003.eqsin.wmnet with OS trixie
  • 16:46 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir6002.*
  • 16:44 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sessionstore1005.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 16:44 tappof@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5 days, 0:00:00 on kafka-logging1003.eqiad.wmnet with reason: migrating to kafka-logging1006
  • 16:36 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir6002.drmrs.wmnet with OS trixie
  • 16:32 jclark@cumin1004: START - Cookbook sre.hosts.provision for host sessionstore1005.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 16:27 jclark@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sessionstore1005.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 16:27 jclark@cumin1004: START - Cookbook sre.hosts.provision for host sessionstore1005.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 16:23 jclark@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sessionstore1005.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 16:22 jclark@cumin1004: START - Cookbook sre.hosts.provision for host sessionstore1005.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 16:16 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 16:15 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add entries for new eqiad links - cmooney@cumin1004"
  • 16:15 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add entries for new eqiad links - cmooney@cumin1004"
  • 16:13 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir6002.drmrs.wmnet with reason: host reimage
  • 16:10 cmooney@cumin1004: START - Cookbook sre.dns.netbox
  • 16:09 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir6002.drmrs.wmnet with reason: host reimage
  • 16:01 cklimas@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply
  • 16:00 cklimas@deploy1003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply
  • 16:00 cklimas@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply
  • 16:00 cklimas@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply
  • 16:00 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host registry2004.codfw.wmnet with OS trixie
  • 15:55 cklimas@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply
  • 15:54 cklimas@deploy1003: helmfile [staging] START helmfile.d/services/wikifeeds: apply
  • 15:49 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
  • 15:45 jdlrobson@deploy1003: Finished scap sync-world: Backport for Fixes: '.action_context' should be string (T437122) (duration: 12m 40s)
  • 15:42 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on registry2004.codfw.wmnet with reason: host reimage
  • 15:39 jdlrobson@deploy1003: jdlrobson: Continuing with deployment
  • 15:39 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir6002.drmrs.wmnet with OS trixie
  • 15:38 elukey@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on registry2004.codfw.wmnet with reason: host reimage
  • 15:36 jdlrobson@deploy1003: jdlrobson: Backport for Fixes: '.action_context' should be string (T437122) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 15:33 slyngshede@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply
  • 15:32 slyngshede@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply
  • 15:32 jdlrobson@deploy1003: Started scap sync-world: Backport for Fixes: '.action_context' should be string (T437122)
  • 15:19 elukey@puppetserver1001: conftool action : set/pooled=false; selector: name=registry2004.*
  • 15:18 elukey@cumin1004: START - Cookbook sre.hosts.reimage for host registry2004.codfw.wmnet with OS trixie
  • 15:16 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir3006.*
  • 15:11 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
  • 15:07 slyngshede@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-web: apply
  • 15:07 slyngshede@deploy1003: helmfile [codfw] START helmfile.d/services/mw-web: apply
  • 15:03 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir3006.esams.wmnet with OS trixie
  • 15:01 slyngshede@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply
  • 15:01 slyngshede@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-ext: apply
  • 14:47 elukey@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host ml-serve1016.eqiad.wmnet with OS trixie
  • 14:39 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir3006.esams.wmnet with reason: host reimage
  • 14:36 elukey@cumin1004: START - Cookbook sre.hosts.reimage for host ml-serve1016.eqiad.wmnet with OS trixie
  • 14:34 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir3006.esams.wmnet with reason: host reimage
  • 14:26 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-serve1016.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 14:20 elukey@cumin1004: START - Cookbook sre.hosts.provision for host ml-serve1016.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 14:13 zabe@deploy1003: Finished scap sync-world: Backport for Move wbc_entity_usage to x1 for mediawikiwiki (T438716), Set db explicitly to false for virtual-wikibase-entityusage (duration: 08m 09s)
  • 14:08 zabe@deploy1003: zabe: Continuing with deployment
  • 14:08 zabe@deploy1003: zabe: Backport for Move wbc_entity_usage to x1 for mediawikiwiki (T438716), Set db explicitly to false for virtual-wikibase-entityusage synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 14:07 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir3006.esams.wmnet with OS trixie
  • 14:05 zabe@deploy1003: Started scap sync-world: Backport for Move wbc_entity_usage to x1 for mediawikiwiki (T438716), Set db explicitly to false for virtual-wikibase-entityusage
  • 14:01 zabe@deploy1003: Started scap sync-world: Backport for Move wbc_entity_usage to x1 for mediawikiwiki (T438716), Set db explicitly to false for virtual-wikibase-entityusage
  • 13:55 dreamyjazz@deploy1003: Finished scap sync-world: Backport for nlwiki: enable SecurePoll local elections (T434045) (duration: 12m 30s)
  • 13:51 dreamyjazz@deploy1003: dreamyjazz, novemlinguae: Continuing with deployment
  • 13:47 dreamyjazz@deploy1003: dreamyjazz, novemlinguae: Backport for nlwiki: enable SecurePoll local elections (T434045) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 13:45 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 13:45 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add entries for new eqiad links - cmooney@cumin1004"
  • 13:45 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add entries for new eqiad links - cmooney@cumin1004"
  • 13:43 zabe: reconcile wbc_entity_usage from local cluster to x1 for mediawikiwiki # T438716
  • 13:43 dreamyjazz@deploy1003: Started scap sync-world: Backport for nlwiki: enable SecurePoll local elections (T434045)
  • 13:41 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-pageview: apply
  • 13:41 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-pageview: apply
  • 13:41 cmooney@cumin1004: START - Cookbook sre.dns.netbox
  • 13:40 samtar@deploy1003: Finished scap sync-world: Backport for arywiki: Create patroller and autopatrolled user groups (T438421) (duration: 11m 40s)
  • 13:36 samtar@deploy1003: samtar, tryvix1509: Continuing with deployment
  • 13:33 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply
  • 13:33 samtar@deploy1003: samtar, tryvix1509: Backport for arywiki: Create patroller and autopatrolled user groups (T438421) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 13:29 samtar@deploy1003: Started scap sync-world: Backport for arywiki: Create patroller and autopatrolled user groups (T438421)
  • 13:22 mfossati@deploy1003: Finished scap sync-world: Backport for Let AA measure eligible readers w/o beta opt-in (T437076) (duration: 14m 19s)
  • 13:15 mfossati@deploy1003: mfossati: Continuing with deployment
  • 13:14 mfossati@deploy1003: mfossati: Backport for Let AA measure eligible readers w/o beta opt-in (T437076) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
  • 13:10 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudvirt1063.eqiad.wmnet
  • 13:07 mfossati@deploy1003: Started scap sync-world: Backport for Let AA measure eligible readers w/o beta opt-in (T437076)
  • 13:01 brouberol@cumin1003: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM archiva1002.wikimedia.org
  • 12:59 filippo@cumin1004: START - Cookbook sre.hosts.reboot-single for host cloudvirt1063.eqiad.wmnet
  • 12:57 brouberol@cumin1003: START - Cookbook sre.ganeti.reboot-vm for VM archiva1002.wikimedia.org
  • 12:54 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply
  • 12:54 jclark@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ml-serve1016.eqiad.wmnet with OS trixie
  • 12:54 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply
  • 12:53 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-eqiad@eqiad
  • 12:53 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs
  • 12:51 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs
  • 12:48 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
  • 12:48 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad
  • 12:48 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
  • 12:36 XioNoX: delete BGP sessions to 15305 in Equinix Ashburn (peer leaving the IX)
  • 12:30 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply
  • 12:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
  • 12:28 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-codfw@codfw
  • 12:28 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs
  • 12:27 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs
  • 12:23 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-codfw@codfw
  • 12:05 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2005.codfw.wmnet
  • 11:45 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/analytics-test: apply
  • 11:45 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/analytics-test: apply
  • 11:45 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wmde: apply
  • 11:45 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2005.codfw.wmnet
  • 11:44 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wmde: apply
  • 11:44 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wikidata: apply
  • 11:44 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wikidata: apply
  • 11:44 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
  • 11:44 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
  • 11:44 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-sre: apply
  • 11:43 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-sre: apply
  • 11:43 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-search: apply
  • 11:42 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-search: apply
  • 11:42 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-research: apply
  • 11:42 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-research: apply
  • 11:42 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-platform-eng: apply
  • 11:41 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-platform-eng: apply
  • 11:41 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
  • 11:40 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply
  • 11:40 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply
  • 11:40 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply
  • 11:40 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-fr-tech: apply
  • 11:40 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2004.codfw.wmnet
  • 11:39 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-fr-tech: apply
  • 11:39 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-experiment-platform: apply
  • 11:38 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-experiment-platform: apply
  • 11:38 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
  • 11:38 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
  • 11:38 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply
  • 11:37 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply
  • 11:37 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-product: apply
  • 11:37 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-product: apply
  • 11:37 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/blunderbuss: apply
  • 11:35 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/blunderbuss: apply
  • 11:35 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthbook: apply
  • 11:35 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook: apply
  • 11:35 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply
  • 11:34 jclark@cumin1004: START - Cookbook sre.hosts.reimage for host ml-serve1016.eqiad.wmnet with OS trixie
  • 11:34 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply
  • 11:34 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/superset: apply
  • 11:33 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/superset: apply
  • 11:33 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/superset-next: apply
  • 11:33 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/superset-next: apply
  • 11:33 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/spark-history: apply
  • 11:32 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/spark-history: apply
  • 11:32 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/spark-history: apply
  • 11:31 jclark@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ml-serve1016.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 11:31 jclark@cumin1004: START - Cookbook sre.hosts.provision for host ml-serve1016.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 11:31 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/spark-history: apply
  • 11:13 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2004.codfw.wmnet
  • 11:13 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2003.codfw.wmnet
  • 11:04 urbanecm@deploy1003: mwscript-k8s job started: extensions/Translate/scripts/moveTranslatableBundle.php --wiki mediawikiwiki 'Wikimedia Apps/Team/Android/Customizable Donation Reminder Experiment' 'Wikimedia Apps/Team/Customizable Donation Reminder/Android' 'Martin Urbanec' --reason 'per request T438704'
  • 10:59 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2003.codfw.wmnet
  • 10:54 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2002.codfw.wmnet
  • 10:50 urbanecm@deploy1003: mwscript-k8s job started: extensions/Translate/scripts/moveTranslatableBundle.php --wiki mediawikiwiki 'Wikimedia Apps/Team/Android/Customizable Donation Reminder Experiment' 'Wikimedia Apps/Team/Customizable Donation Reminder/Android' Zabe --reason 'per request T438704'
  • 10:38 zabe: create wbc_entity_usage table in x1 for all wikidata client wikis # T438499
  • 10:36 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2002.codfw.wmnet
  • 10:36 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2001.codfw.wmnet
  • 10:21 zabe@deploy1003: mwscript-k8s job started: extensions/Translate/scripts/moveTranslatableBundle.php --wiki mediawikiwiki 'Wikimedia Apps/Team/Android/Customizable Donation Reminder Experiment' 'Wikimedia Apps/Team/Customizable Donation Reminder/Android' Zabe --reason 'per request T438704'
  • 10:21 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2001.codfw.wmnet
  • 10:21 zabe@deploy1003: mwscript-k8s job started: extensions/Translate/scripts/moveTranslatableBundle.php --wiki mediawikiwiki 'Wikimedia Apps/Team/Android/Customizable Donation Reminder Experiment' 'Wikimedia Apps/Team/Customizable Donation Reminder/Android' Zabe --reason 'per request T438704'
  • 10:20 zabe@deploy1003: mwscript-k8s job started: extensions/Translate/scripts/moveTranslatableBundle.php --wiki metawiki 'Wikimedia Apps/Team/Android/Customizable Donation Reminder Experiment' 'Wikimedia Apps/Team/Customizable Donation Reminder/Android' Zabe --reason 'per request T438704'
  • 10:17 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-eqiad@eqiad
  • 10:17 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs
  • 10:16 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs
  • 10:12 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/proton: apply
  • 10:11 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad
  • 10:09 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/proton: apply
  • 10:04 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/proton: apply
  • 10:02 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/proton: apply
  • 10:01 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/proton: apply
  • 10:00 jmm@deploy1003: helmfile [staging] START helmfile.d/services/proton: apply
  • 10:00 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/proton: apply
  • 09:59 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1063.eqiad.wmnet with OS trixie
  • 09:59 jmm@deploy1003: helmfile [staging] START helmfile.d/services/proton: apply
  • 09:56 klausman@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: apply
  • 09:55 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-codfw@codfw
  • 09:55 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs
  • 09:54 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs
  • 09:54 klausman@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: apply
  • 09:50 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-codfw@codfw
  • 09:35 moritzm: installing chromium security updates
  • 09:22 tappof: bump space for prometheus k8s-dse in eqiad
  • 09:11 ihurbain@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
  • 09:07 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1063.eqiad.wmnet with reason: host reimage
  • 09:04 ihurbain@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
  • 09:04 ihurbain@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
  • 09:01 urbanecm@deploy1003: Finished scap sync-world: Backport for [Growth] Remove unused config variables (T392944) (duration: 32m 54s)
  • 09:01 filippo@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1063.eqiad.wmnet with reason: host reimage
  • 08:58 ihurbain@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
  • 08:45 filippo@cumin1004: START - Cookbook sre.hosts.reimage for host cloudvirt1063.eqiad.wmnet with OS trixie
  • 08:29 urbanecm@deploy1003: Started scap sync-world: Backport for [Growth] Remove unused config variables (T392944)
  • 08:15 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudvirt1063.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 08:05 filippo@cumin1004: START - Cookbook sre.hosts.provision for host cloudvirt1063.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 08:04 filippo@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on cloudvirt1063.eqiad.wmnet with reason: provision
  • 08:01 XioNoX: restart gnmic on all netflow servers except 2005 and 1004 to pickup the new version - T438291
  • 07:59 XioNoX: install gnmic 0.49 on all netflow hosts - T438291
  • 07:57 XioNoX: add gnmic 0.49 to trixie-wikimedia - T438291
  • 07:53 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device fasw1-f5a-codfw
  • 07:53 ayounsi@cumin1004: START - Cookbook sre.network.tls for network device fasw1-f5a-codfw
  • 07:53 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device fasw1-f5b-codfw
  • 07:53 ayounsi@cumin1004: START - Cookbook sre.network.tls for network device fasw1-f5b-codfw
  • 07:45 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudvirt1077.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 07:39 filippo@cumin1004: START - Cookbook sre.hosts.provision for host cloudvirt1077.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
  • 07:37 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudvirt1077.eqiad.wmnet
  • 07:23 filippo@cumin1004: START - Cookbook sre.hosts.reboot-single for host cloudvirt1077.eqiad.wmnet
  • 07:13 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'.
  • 07:12 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'.
  • 07:11 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
  • 07:10 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
  • 07:00 jmm@cumin2003: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for krb1002.eqiad.wmnet: Renew puppet certificate - jmm@cumin2003
  • 05:24 moritzm: upgrade docker-report on build2004 to 0.0.20 T435314
  • 05:14 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast1004.wikimedia.org

2026-09-20

  • 20:08 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
  • 20:08 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
  • 20:08 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
  • 20:08 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
  • 20:08 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
  • 20:07 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply

2026-09-19

  • 16:55 ssastry@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
  • 16:55 ssastry@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
  • 16:55 ssastry@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
  • 16:55 ssastry@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
  • 14:11 urbanecm: Attach SHB@commonswiki to the SUL account manually (T438591, see T438591#12341750 for what I did exactly)
  • 04:08 ssastry@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
  • 04:08 ssastry@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
  • 04:08 ssastry@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
  • 04:07 ssastry@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
  • 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 36s)
  • 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image

Other archives

See Server Admin Log/Archives.

Retrieved from "https://wikitech.wikimedia.org/w/index.php?title=Server_Admin_Log&oldid=2475918"