GH action for updating logstash version #18035

donoghuc · 2025-08-25T22:17:37Z

This commit adds a GH action for bumping the logstash version. It uses the updatescli based on a suggestion for how the beats team is doing file updates. There is a workflow_dispatch trigger that accepts a new logstash version. Both the versions.yml and lock file are updated. If it is a branch where we are not vendoring a lockfile that file is skipped without failing the workflow step.

Implements https://github.com/elastic/ingest-dev/issues/5977

github-actions · 2025-08-25T22:17:47Z

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

mergify · 2025-08-25T22:18:15Z

This pull request does not have a backport label. Could you fix it @donoghuc? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit.
If no backport is necessary, please add the backport-skip label

donoghuc · 2025-08-25T22:18:20Z

Use of updatecli for dep bumps was based on this comment #17945 (comment)

donoghuc · 2025-08-25T22:22:47Z

To test locally:

install updatecli

brew install updatecli/updatecli/updatecli

generate config file:

# Create the same config the GitHub Action uses
cat > test-updatecli.yml << 'EOF'
name: Update versions.yml

sources:
  logstash_version:
    kind: shell
    spec:
      command: echo "9.3.0"

targets:
  update_logstash_version:
    name: Update logstash version in versions.yml
    kind: yaml
    spec:
      file: versions.yml
      key: $.logstash
    sourceid: logstash_version
  
  update_logstash_core_version:
    name: Update logstash-core version in versions.yml
    kind: yaml
    spec:
      file: versions.yml
      key: $.logstash-core
    sourceid: logstash_version

---
name: Update lock file

sources:
  logstash_version:
    kind: shell
    spec:
      command: echo "9.3.0"

conditions:
  check_lockfile_exists:
    name: Check if lockfile exists
    kind: shell
    disablesourceinput: true
    spec:
      command: test -f Gemfile.jruby-3.1.lock.release

targets:
  update_gemfile_lock_dependency:
    name: Update logstash-core dependency
    kind: file
    spec:
      file: Gemfile.jruby-3.1.lock.release
      matchpattern: 'logstash-core \(= [0-9]+\.[0-9]+\.[0-9]+'
      replacepattern: 'logstash-core (= 9.3.0'
    sourceid: logstash_version
  
  update_gemfile_lock_spec:
    name: Update logstash-core spec
    kind: file
    spec:
      file: Gemfile.jruby-3.1.lock.release
      matchpattern: 'logstash-core \([0-9]+\.[0-9]+\.[0-9]+-java\)'
      replacepattern: 'logstash-core (9.3.0-java)'
    sourceid: logstash_version
EOF

run

➜  logstash git:(bump-ls-version-action) ✗ updatecli apply --config test-updatecli.yml



+++++++++++
+ PREPARE +
+++++++++++

Loading Pipeline "test-updatecli.yml"

SCM repository retrieved: 0


++++++++++++++++++
+ AUTO DISCOVERY +
++++++++++++++++++



++++++++++++
+ PIPELINE +
++++++++++++



#######################
# UPDATE VERSIONS.YML #
#######################

source: source#logstash_version
-----------------------
The shell 🐚 command "/bin/sh /var/folders/cw/q_xjr4md1wj_w_c1xwfrnxdw0000gn/T/updatecli/bin/33b3af0384bd90f14aef112598cfabdf6d6e77168b475723abbaaf5f269b28f2.sh" ran successfully with the following output:
----
9.3.0
----
✔ shell command executed successfully

target: target#update_logstash_core_version
-----------------------------------
⚠ - change detected:
	* key "$.logstash-core" updated from "9.2.0" to "9.3.0", in file "versions.yml"

target: target#update_logstash_version
------------------------------
⚠ - change detected:
	* key "$.logstash" updated from "9.2.0" to "9.3.0", in file "versions.yml"


####################
# UPDATE LOCK FILE #
####################

source: source#logstash_version
-----------------------
The shell 🐚 command "/bin/sh /var/folders/cw/q_xjr4md1wj_w_c1xwfrnxdw0000gn/T/updatecli/bin/33b3af0384bd90f14aef112598cfabdf6d6e77168b475723abbaaf5f269b28f2.sh" ran successfully with the following output:
----
9.3.0
----
✔ shell command executed successfully

condition: condition#check_lockfile_exists
-------------------------------
The shell 🐚 command "/bin/sh /var/folders/cw/q_xjr4md1wj_w_c1xwfrnxdw0000gn/T/updatecli/bin/c4efa96ffd6665bfcc18724aff4523a9b7e976d0a59d6da437e1263a76b5f8c6.sh" exited on error (exit code 1) with the following output:
----
----

command stderr output was:
----
----
shell command failed. Expected exit code 0 but got 1
✗ shell condition of type "console/output" not passing

target: target#update_gemfile_lock_spec
-------------------------------

target: target#update_gemfile_lock_dependency
-------------------------------------


ACTIONS
========


=============================

SUMMARY:



- Update lock file:
	Source:
		✔ [logstash_version]
	Condition:
		✗ [check_lockfile_exists] Check if lockfile exists
	Target:
		- [update_gemfile_lock_dependency] Update logstash-core dependency
		- [update_gemfile_lock_spec] Update logstash-core spec


⚠ Update versions.yml:
	Source:
		✔ [logstash_version]
	Target:
		⚠ [update_logstash_core_version] Update logstash-core version in versions.yml
		⚠ [update_logstash_version] Update logstash version in versions.yml


Run Summary
===========
Pipeline(s) run:
  * Changed:	1
  * Failed:	0
  * Skipped:	1
  * Succeeded:	0
  * Total:	2

donoghuc · 2025-08-27T23:10:08Z

This was tested with #18050 and produced a PR with a version bump! Ready for review.

.github/workflows/bump-logstash.yml

v1v · 2025-09-09T06:55:31Z

.ci/updatecli/values.d/scm.yml

+  # begin updatecli-compose policy values
+  user: 'github-actions[bot]'
+  email: '41898282+github-actions[bot]@users.noreply.github.com'
+  # end updatecli-compose policy values


You don't need those values, that's when using a shared policy with updatecli, in this case, you only added the .ci/updatecli/bump-logstash-version.yml manifest.

Suggested change

# begin updatecli-compose policy values

user: 'github-actions[bot]'

email: '41898282+github-actions[bot]@users.noreply.github.com'

# end updatecli-compose policy values

v1v · 2025-09-09T06:56:55Z

.github/workflows/bump-java-version.yml

@@ -1,19 +1,34 @@
-name: Stub GH action for devoping new workflows [STUB]
+


duplicates .github/workflows/bump-version.yml?

same comment as #18035 (comment)

v1v · 2025-09-09T06:57:49Z

.github/workflows/bump-java-version.yml

  pull_request:
    types: [opened, synchronize, reopened]


IIUC, this event will not work with forked PRs, since the GH token will never have any elevated access but read-only.

Yeah, the idea here was that this "stub" action is already checked in with a pull_request trigger. I was hoping that by doing that and targeting that workflow file from a branch that pushed to the elastic master copy of logstash would be enough to actually give a valid token for raising a PR. It may not actually be necessary though. Either way I will be removing the "stub" before this is actually ready for review.

This commit adds a GH action for bumping the logstash version. It uses the updatescli based on a suggestion for how the beats team is doing file updates. There is a workflow_dispatch trigger that accepts a new logstash version. Both the versions.yml and lock file are updated. If it is a branch where we are not vendoring a lockfile that file is skipped without failing the workflow step.

use pattern suggested in code review

v1v · 2025-09-10T09:13:42Z

.ci/updatecli/bump-logstash-version-lockfile.yml

@@ -0,0 +1,47 @@
+---
+name: Update logstash lockfile
+pipelineid: "logstash/version-updates"


If you want to use this updatecli manifest to target multiple branches, then use {{ requiredEnv "LOGSTASH_BRANCH" }} in tghe pipelineid, that's the name of the base branch when a PR is created

Suggested change

pipelineid: "logstash/version-updates"

pipelineid: "logstash/version-updates-{{ requiredEnv "LOGSTASH_BRANCH" }}"

For sure open to this change, but i'm not quite following where this is used in PR branching. In #18163 it seems like the branch is updatecli_8.18_logstash/version-updates. If somehow this was not auto-deleted on merged I could see there being a conflict (and thus probably not a PR raised).

.github/workflows/bump-logstash.yml

donoghuc · 2025-09-10T23:18:14Z

Thanks @v1v for taking a look. I was working yesterday on figuring out how to implement the following using updatecli:

For unreleased branches (for example main or eventually a 9.x we wont vendor a lock file (and therefore dont need to update it). However, we will always want to update versions.yml. I want a workflow where a single PR is raised where versions.yml is always updated and the lock file is only updated if it exists.

So far i cant figure out how to express this in updatecli. Is that something you have an idea of how to do?

v1v · 2025-09-11T07:09:54Z

So far i cant figure out how to express this in updatecli. Is that something you have an idea of how to do?

I guess you can use the conditionals with sources and targets.

The example below will search for the given file.lock and update it, in addition, it modifies the versions.yml regardless.

sources:
  lock_file_exists:
    kind: shell
    spec:
      command: test -f file.lock

targets:
  update_lock_file:
    disablesourceinput: true
    dependson:
      - 'source#lock_file_exists'
    kind: file
    spec:
      file: "file.lock"
      content: "foo1"

  update_versions.yml:
    kind: file
    disablesourceinput: true
    spec:
      file: "versions.yml"
      content: "bar1"

If lock file does not exist

	Source:
		✗ [lock_file_exists] 
	Target:
		- [update_lock_file] 
		⚠ [update_versions.yml]

Otherwise:

	Source:
		✔ [lock_file_exists] 
	Target:
		⚠ [update_lock_file] 
		⚠ [update_versions.yml]

mergify · 2025-09-12T21:18:50Z

⚠️ The sha of the head commit of this PR conflicts with #18050. Mergify cannot evaluate rules on this PR. ⚠️

donoghuc · 2025-09-12T21:53:47Z

Thanks so much @v1v ! I think i've got this to a place where its working. Here is a run against a branch with a lockfile #18163 and here is one without #18162

Really appreciate your help on this, this tooling is great!

v1v · 2025-09-16T19:23:09Z

.ci/updatecli/bump-logstash-version.yml

@@ -0,0 +1,81 @@
+---
+name: Update logstash version files
+pipelineid: "logstash/version-updates"


If you run this manifest targeting multiple branches, then the pipelineid should be unique. I'd recommend using LOGSTASH_BRANCH.

Suggested change

pipelineid: "logstash/version-updates"

pipelineid: "logstash/version-updates-{{ requiredEnv "LOGSTASH_BRANCH" }}"

v1v · 2025-09-16T19:24:42Z

.github/workflows/bump-logstash.yml

@@ -0,0 +1,31 @@
+name: bump-logstash-version


IIUC, .github/workflows/bump-java-version.yml and .github/workflows/bump-logstash.yml do the same, don't they?

See #18035 (comment)

I've reverted all the changes to that file. I'll deal with it separately.

v1v · 2025-09-16T19:25:13Z

.ci/updatecli/values.d/scm.yml

+  enabled: true
+  owner: elastic
+  repository: logstash
+  commitusingapi: true


nit: this is hardcoded in the updatecli manifest

Suggested change

commitusingapi: true

v1v · 2025-09-16T19:25:38Z

.ci/updatecli/values.d/scm.yml

@@ -0,0 +1,5 @@
+scm:
+  enabled: true


nit: unused

Suggested change

enabled: true

v1v · 2025-09-16T19:26:11Z

.github/workflows/bump-java-version.yml

 on:
  workflow_dispatch:
+    inputs:
+      logstash_version:
+        description: 'Logstash version'


Do you think adding an example might be useful here?

Suggested change

description: 'Logstash version'

description: 'Logstash version (i.e: 1.2.3)'

yep, added one

v1v · 2025-09-16T19:26:34Z

.github/workflows/bump-java-version.yml

+        required: true
+        type: string
+      logstash_branch:
+        description: 'Logstash branch'


Suggested change

description: 'Logstash branch'

description: 'Logstash branch (i.e: 8.19)'

yep, added one

elastic-sonarqube · 2025-09-16T22:26:09Z

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarQube

elasticmachine · 2025-09-16T22:34:04Z

💚 Build Succeeded

Buildkite Build
Commit: ea5ddcc

History

💚 Build #3509 succeeded 364c831
💚 Build #3498 succeeded fa1538e
💚 Build #3497 succeeded f6306aa
💛 Build #3491 was flaky fe19127
💚 Build #3484 succeeded 83568e4
💛 Build #3463 was flaky 5118e44

donoghuc mentioned this pull request Aug 26, 2025

Replace buildkite jdk version check w/GH action #17945

Open

5 tasks

donoghuc force-pushed the bump-ls-version-action branch from 410469e to 2514eec Compare August 27, 2025 22:37

mergify bot mentioned this pull request Aug 27, 2025

TEST: use stub GH action workflow to test new LS version action [NOT FOR MERGE] #18050

Open

donoghuc force-pushed the bump-ls-version-action branch 3 times, most recently from 0c6def8 to 1292374 Compare August 27, 2025 23:09

donoghuc marked this pull request as ready for review August 27, 2025 23:10

donoghuc requested review from a team as code owners August 27, 2025 23:10

v1v reviewed Sep 2, 2025

View reviewed changes

.github/workflows/bump-logstash.yml Outdated Show resolved Hide resolved

.github/workflows/bump-logstash.yml Outdated Show resolved Hide resolved

donoghuc force-pushed the bump-ls-version-action branch 2 times, most recently from 956fd79 to 38c0131 Compare September 8, 2025 22:35

v1v reviewed Sep 9, 2025

View reviewed changes

donoghuc added 12 commits September 9, 2025 14:56

use stub workflow to test

e75a275

Separate step when modifying

9f2a3bd

dont track temp updatecli file and install action

3d29a0a

restore stub

fafaf94

Refactor updatecli workflow

beab304

use pattern suggested in code review

Use test stub to validate changes

537f00d

fix inputs section

f7bf2c6

fix branch

5eb366e

dont fail if file is not present

c0b7aab

try with no global conditionals

bc2f8da

file condition

76da246

donoghuc added 5 commits September 9, 2025 14:56

name conditional

65baa46

split into separate pipeline files

d53ef1d

remove unused scmid

ec98a22

no explicit pr action?

9eff1f1

add back PR

83568e4

donoghuc force-pushed the bump-ls-version-action branch from 5118e44 to 83568e4 Compare September 9, 2025 22:30

v1v reviewed Sep 10, 2025

View reviewed changes

donoghuc added 3 commits September 12, 2025 13:25

try sources pattern

fe19127

explicitly connect target with scmid

5a3ba90

ensure lock file lookup executes from checkout

f6306aa

Cleanup

fa1538e

donoghuc added 2 commits September 16, 2025 10:35

test

c7bc793

restore LOGSTASH_BRANCH

364c831

v1v reviewed Sep 16, 2025

View reviewed changes

codereview feedback

ea5ddcc

donoghuc requested a review from v1v September 16, 2025 22:12

		@@ -1,19 +1,34 @@
		name: Stub GH action for devoping new workflows [STUB]

	pipelineid: "logstash/version-updates"
	pipelineid: "logstash/version-updates-{{ requiredEnv "LOGSTASH_BRANCH" }}"

	description: 'Logstash version'
	description: 'Logstash version (i.e: 1.2.3)'

	description: 'Logstash branch'
	description: 'Logstash branch (i.e: 8.19)'

GH action for updating logstash version #18035

Are you sure you want to change the base?

GH action for updating logstash version #18035

Conversation

donoghuc commented Aug 25, 2025

Uh oh!

github-actions bot commented Aug 25, 2025

🤖 GitHub comments

Uh oh!

mergify bot commented Aug 25, 2025

Uh oh!

donoghuc commented Aug 25, 2025

Uh oh!

donoghuc commented Aug 25, 2025

Uh oh!

donoghuc commented Aug 27, 2025

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

donoghuc commented Sep 10, 2025

Uh oh!

v1v commented Sep 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mergify bot commented Sep 12, 2025

Uh oh!

donoghuc commented Sep 12, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

elastic-sonarqube bot commented Sep 16, 2025

Quality Gate passed

Uh oh!

elasticmachine commented Sep 16, 2025

💚 Build Succeeded

History

Uh oh!

Uh oh!

v1v commented Sep 11, 2025 •

edited

Loading