Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

plugin manager: add --level=[major|minor|patch] (default: minor) #16899

Open
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

yaauie
Copy link
Member

@yaauie yaauie commented Jan 14, 2025

Release notes

  • By default, the plugin manager will no longer consume breaking changes in plugins, and will instead only consume minor revisions. If you wish to consume breaking changes, you must now explicitly add the --level=major flag.

What does this PR do?

Adds --level=[major|minor|patch] to the bin/logstash-plugin update command, with default value minor, and propagates it to the underlying bundler's --major, --minor, or --patch flags, to ensure users can upgrade while avoiding breaking changes.

Why is it important/What is the impact to the user?

Users are currently being surprised when their bin/logstash-plugin update consumes breaking changes in plugins, causing their currently-valid pipeline configurations to fail to load. This removes the element of surprise, while still allowing users to upgrade across the major version of the plugins if they expressly wish to do so.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • [ ] I have made corresponding changes to the documentation
  • [ ] I have made corresponding change to the default configuration files (and/or docker env variables)
  • I have added tests that prove my fix is effective or that my feature works

closes: #16894

@jsvd
Copy link
Member

jsvd commented Jan 14, 2025

Confirmed it works correctly:

Vanilla 8.17.0 `bin/logstash-plugin update`
/tmp/logstash-8.17.0
❯ bin/logstash-plugin update
Using bundled JDK: /tmp/logstash-8.17.0/jdk.app/Contents/Home
Updating logstash-codec-avro, logstash-codec-cef, logstash-codec-collectd, logstash-codec-dots, logstash-codec-edn, logstash-codec-edn_lines, logstash-codec-es_bulk, logstash-codec-fluent, logstash-codec-graphite, logstash-codec-json, logstash-codec-json_lines, logstash-codec-line, logstash-codec-msgpack, logstash-codec-multiline, logstash-codec-netflow, logstash-codec-plain, logstash-codec-rubydebug, logstash-filter-aggregate, logstash-filter-anonymize, logstash-filter-cidr, logstash-filter-clone, logstash-filter-csv, logstash-filter-date, logstash-filter-de_dot, logstash-filter-dissect, logstash-filter-dns, logstash-filter-drop, logstash-filter-elastic_integration, logstash-filter-elasticsearch, logstash-filter-fingerprint, logstash-filter-geoip, logstash-filter-grok, logstash-filter-http, logstash-filter-json, logstash-filter-kv, logstash-filter-memcached, logstash-filter-metrics, logstash-filter-mutate, logstash-filter-prune, logstash-filter-ruby, logstash-filter-sleep, logstash-filter-split, logstash-filter-syslog_pri, logstash-filter-throttle, logstash-filter-translate, logstash-filter-truncate, logstash-filter-urldecode, logstash-filter-useragent, logstash-filter-uuid, logstash-filter-xml, logstash-input-azure_event_hubs, logstash-input-beats, logstash-input-couchdb_changes, logstash-input-dead_letter_queue, logstash-input-elastic_serverless_forwarder, logstash-input-elasticsearch, logstash-input-exec, logstash-input-file, logstash-input-ganglia, logstash-input-gelf, logstash-input-generator, logstash-input-graphite, logstash-input-heartbeat, logstash-input-http, logstash-input-http_poller, logstash-input-jms, logstash-input-pipe, logstash-input-redis, logstash-input-stdin, logstash-input-syslog, logstash-input-tcp, logstash-input-twitter, logstash-input-udp, logstash-input-unix, logstash-integration-aws, logstash-integration-elastic_enterprise_search, logstash-integration-jdbc, logstash-integration-kafka, logstash-integration-logstash, logstash-integration-rabbitmq, logstash-integration-snmp, logstash-output-csv, logstash-output-elasticsearch, logstash-output-email, logstash-output-file, logstash-output-graphite, logstash-output-http, logstash-output-lumberjack, logstash-output-nagios, logstash-output-null, logstash-output-pipe, logstash-output-redis, logstash-output-stdout, logstash-output-tcp, logstash-output-udp, logstash-output-webhdfs
WARN: Unresolved or ambiguous specs during Gem::Specification.reset:
    date (>= 0)
    Available/installed versions of this gem:
    - 3.4.0
    - 3.3.3
WARN: Clearing out unresolved specs. Try 'gem cleanup <gem>'
Please report a bug if this causes problems.
Bundler attempted to update logstash-mixin-ecs_compatibility_support but its version stayed the same
Bundler attempted to update logstash-mixin-event_support but its version stayed the same
Bundler attempted to update logstash-mixin-validator_support but its version stayed the same
Bundler attempted to update logstash-mixin-ca_trusted_fingerprint_support but its version stayed the same
Bundler attempted to update logstash-mixin-normalize_config_support but its version stayed the same
Bundler attempted to update logstash-mixin-http_client but its version stayed the same
Bundler attempted to update logstash-mixin-deprecation_logger_support but its version stayed the same
Bundler attempted to update logstash-mixin-scheduler but its version stayed the same
Bundler attempted to update logstash-mixin-plugin_factory_support but its version stayed the same
Bundler attempted to update logstash-codec-avro but its version stayed the same
Bundler attempted to update logstash-codec-cef but its version stayed the same
Bundler attempted to update logstash-codec-collectd but its version stayed the same
Bundler attempted to update logstash-codec-dots but its version stayed the same
Bundler attempted to update logstash-codec-edn but its version stayed the same
Bundler attempted to update logstash-codec-edn_lines but its version stayed the same
Bundler attempted to update logstash-codec-es_bulk but its version stayed the same
Bundler attempted to update logstash-codec-fluent but its version stayed the same
Bundler attempted to update logstash-codec-graphite but its version stayed the same
Bundler attempted to update logstash-codec-json but its version stayed the same
Bundler attempted to update logstash-codec-json_lines but its version stayed the same
Bundler attempted to update logstash-codec-line but its version stayed the same
Bundler attempted to update logstash-codec-msgpack but its version stayed the same
Bundler attempted to update logstash-codec-multiline but its version stayed the same
Bundler attempted to update logstash-codec-netflow but its version stayed the same
Bundler attempted to update logstash-codec-plain but its version stayed the same
Bundler attempted to update logstash-codec-rubydebug but its version stayed the same
Bundler attempted to update logstash-filter-aggregate but its version stayed the same
Bundler attempted to update logstash-filter-anonymize but its version stayed the same
Bundler attempted to update logstash-filter-cidr but its version stayed the same
Bundler attempted to update logstash-filter-clone but its version stayed the same
Bundler attempted to update logstash-filter-csv but its version stayed the same
Bundler attempted to update logstash-filter-date but its version stayed the same
Bundler attempted to update logstash-filter-de_dot but its version stayed the same
Bundler attempted to update logstash-filter-dissect but its version stayed the same
Bundler attempted to update logstash-filter-dns but its version stayed the same
Bundler attempted to update logstash-filter-drop but its version stayed the same
Bundler attempted to update logstash-filter-fingerprint but its version stayed the same
Bundler attempted to update logstash-filter-geoip but its version stayed the same
Bundler attempted to update logstash-filter-grok but its version stayed the same
Bundler attempted to update logstash-filter-json but its version stayed the same
Bundler attempted to update logstash-filter-kv but its version stayed the same
Bundler attempted to update logstash-filter-memcached but its version stayed the same
Bundler attempted to update logstash-filter-metrics but its version stayed the same
Bundler attempted to update logstash-filter-mutate but its version stayed the same
Bundler attempted to update logstash-filter-prune but its version stayed the same
Bundler attempted to update logstash-filter-ruby but its version stayed the same
Bundler attempted to update logstash-filter-sleep but its version stayed the same
Bundler attempted to update logstash-filter-split but its version stayed the same
Bundler attempted to update logstash-filter-syslog_pri but its version stayed the same
Bundler attempted to update logstash-filter-throttle but its version stayed the same
Bundler attempted to update logstash-filter-translate but its version stayed the same
Bundler attempted to update logstash-filter-truncate but its version stayed the same
Bundler attempted to update logstash-filter-urldecode but its version stayed the same
Bundler attempted to update logstash-filter-useragent but its version stayed the same
Bundler attempted to update logstash-filter-uuid but its version stayed the same
Bundler attempted to update logstash-filter-xml but its version stayed the same
Bundler attempted to update logstash-input-couchdb_changes but its version stayed the same
Bundler attempted to update logstash-input-dead_letter_queue but its version stayed the same
Bundler attempted to update logstash-input-exec but its version stayed the same
Bundler attempted to update logstash-input-file but its version stayed the same
Bundler attempted to update logstash-input-ganglia but its version stayed the same
Bundler attempted to update logstash-input-gelf but its version stayed the same
Bundler attempted to update logstash-input-generator but its version stayed the same
Bundler attempted to update logstash-input-graphite but its version stayed the same
Bundler attempted to update logstash-input-heartbeat but its version stayed the same
Bundler attempted to update logstash-input-jms but its version stayed the same
Bundler attempted to update logstash-input-pipe but its version stayed the same
Bundler attempted to update logstash-input-redis but its version stayed the same
Bundler attempted to update logstash-input-stdin but its version stayed the same
Bundler attempted to update logstash-input-syslog but its version stayed the same
Bundler attempted to update logstash-input-twitter but its version stayed the same
Bundler attempted to update logstash-input-udp but its version stayed the same
Bundler attempted to update logstash-input-unix but its version stayed the same
Bundler attempted to update logstash-integration-aws but its version stayed the same
Bundler attempted to update logstash-integration-rabbitmq but its version stayed the same
Bundler attempted to update logstash-output-csv but its version stayed the same
Bundler attempted to update logstash-output-email but its version stayed the same
Bundler attempted to update logstash-output-file but its version stayed the same
Bundler attempted to update logstash-output-graphite but its version stayed the same
Bundler attempted to update logstash-output-lumberjack but its version stayed the same
Bundler attempted to update logstash-output-nagios but its version stayed the same
Bundler attempted to update logstash-output-null but its version stayed the same
Bundler attempted to update logstash-output-pipe but its version stayed the same
Bundler attempted to update logstash-output-redis but its version stayed the same
Bundler attempted to update logstash-output-stdout but its version stayed the same
Bundler attempted to update logstash-output-udp but its version stayed the same
Bundler attempted to update logstash-output-webhdfs but its version stayed the same
Updated logstash-filter-elastic_integration 0.1.16 to 8.17.0
Updated logstash-filter-elasticsearch 3.16.1 to 4.0.0
Updated logstash-filter-http 1.6.0 to 2.0.0
Updated logstash-input-azure_event_hubs 1.5.0 to 1.5.1
Updated logstash-input-beats 6.9.1 to 7.0.0
Updated logstash-input-elastic_serverless_forwarder 0.1.5 to 2.0.0
Updated logstash-input-elasticsearch 4.20.4 to 5.0.0
Updated logstash-input-http 3.9.2 to 4.1.0
Updated logstash-input-http_poller 5.6.0 to 6.0.0
Updated logstash-input-tcp 6.4.4 to 7.0.0
Updated logstash-integration-elastic_enterprise_search 3.0.0 to 3.0.1
Updated logstash-integration-jdbc 5.5.0 to 5.5.2
Updated logstash-integration-kafka 11.5.2 to 11.6.0
Updated logstash-integration-logstash 1.0.3 to 1.0.4
Updated logstash-integration-snmp 4.0.4 to 4.0.5
Updated logstash-output-elasticsearch 11.22.9 to 12.0.0
Updated logstash-output-http 5.7.1 to 6.0.0
Updated logstash-output-tcp 6.2.1 to 7.0.0
Patched v8.17.0 `bin/logstash-plugin update`
~/elastic/logstash tags/v8.17.0* 2m 17s
❯ curl https://patch-diff.githubusercontent.com/raw/elastic/logstash/pull/16899.patch | patch -p1
% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                               Dload  Upload   Total   Spent    Left  Speed
100  8167    0  8167    0     0  16375      0 --:--:-- --:--:-- --:--:-- 16399
patching file lib/bootstrap/bundler.rb
patching file lib/pluginmanager/update.rb
patching file spec/unit/bootstrap/bundler_spec.rb
patching file spec/unit/plugin_manager/update_spec.rb

~/elastic/logstash tags/v8.17.0*
❯ bin/logstash-plugin update
Using system java: /Users/joaoduarte/.sdkman/candidates/java/current/bin/java
Updating logstash-codec-avro, logstash-codec-cef, logstash-codec-collectd, logstash-codec-dots, logstash-codec-edn, logstash-codec-edn_lines, logstash-codec-es_bulk, logstash-codec-fluent, logstash-codec-graphite, logstash-codec-json, logstash-codec-json_lines, logstash-codec-line, logstash-codec-msgpack, logstash-codec-multiline, logstash-codec-netflow, logstash-codec-plain, logstash-codec-rubydebug, logstash-filter-aggregate, logstash-filter-anonymize, logstash-filter-cidr, logstash-filter-clone, logstash-filter-csv, logstash-filter-date, logstash-filter-de_dot, logstash-filter-dissect, logstash-filter-dns, logstash-filter-drop, logstash-filter-elastic_integration, logstash-filter-elasticsearch, logstash-filter-fingerprint, logstash-filter-geoip, logstash-filter-grok, logstash-filter-http, logstash-filter-json, logstash-filter-kv, logstash-filter-memcached, logstash-filter-metrics, logstash-filter-mutate, logstash-filter-prune, logstash-filter-ruby, logstash-filter-sleep, logstash-filter-split, logstash-filter-syslog_pri, logstash-filter-throttle, logstash-filter-translate, logstash-filter-truncate, logstash-filter-urldecode, logstash-filter-useragent, logstash-filter-uuid, logstash-filter-xml, logstash-input-azure_event_hubs, logstash-input-beats, logstash-input-couchdb_changes, logstash-input-dead_letter_queue, logstash-input-elastic_serverless_forwarder, logstash-input-elasticsearch, logstash-input-exec, logstash-input-file, logstash-input-ganglia, logstash-input-gelf, logstash-input-generator, logstash-input-graphite, logstash-input-heartbeat, logstash-input-http, logstash-input-http_poller, logstash-input-jms, logstash-input-pipe, logstash-input-redis, logstash-input-stdin, logstash-input-syslog, logstash-input-tcp, logstash-input-twitter, logstash-input-udp, logstash-input-unix, logstash-integration-aws, logstash-integration-elastic_enterprise_search, logstash-integration-jdbc, logstash-integration-kafka, logstash-integration-logstash, logstash-integration-rabbitmq, logstash-integration-snmp, logstash-output-csv, logstash-output-elasticsearch, logstash-output-email, logstash-output-file, logstash-output-graphite, logstash-output-http, logstash-output-lumberjack, logstash-output-nagios, logstash-output-null, logstash-output-pipe, logstash-output-redis, logstash-output-stdout, logstash-output-tcp, logstash-output-udp, logstash-output-webhdfs
WARN: Unresolved or ambiguous specs during Gem::Specification.reset:
    date (>= 0)
    Available/installed versions of this gem:
    - 3.4.0
    - 3.3.3
WARN: Clearing out unresolved specs. Try 'gem cleanup <gem>'
Please report a bug if this causes problems.
Bundler attempted to update logstash-mixin-ecs_compatibility_support but its version stayed the same
Bundler attempted to update logstash-mixin-event_support but its version stayed the same
Bundler attempted to update logstash-mixin-validator_support but its version stayed the same
Bundler attempted to update logstash-mixin-ca_trusted_fingerprint_support but its version stayed the same
Bundler attempted to update logstash-mixin-normalize_config_support but its version stayed the same
Bundler attempted to update logstash-mixin-http_client but its version stayed the same
Bundler attempted to update logstash-mixin-deprecation_logger_support but its version stayed the same
Bundler attempted to update logstash-mixin-scheduler but its version stayed the same
Bundler attempted to update logstash-mixin-plugin_factory_support but its version stayed the same
Bundler attempted to update logstash-codec-avro but its version stayed the same
Bundler attempted to update logstash-codec-cef but its version stayed the same
Bundler attempted to update logstash-codec-collectd but its version stayed the same
Bundler attempted to update logstash-codec-dots but its version stayed the same
Bundler attempted to update logstash-codec-edn but its version stayed the same
Bundler attempted to update logstash-codec-edn_lines but its version stayed the same
Bundler attempted to update logstash-codec-es_bulk but its version stayed the same
Bundler attempted to update logstash-codec-fluent but its version stayed the same
Bundler attempted to update logstash-codec-graphite but its version stayed the same
Bundler attempted to update logstash-codec-json but its version stayed the same
Bundler attempted to update logstash-codec-json_lines but its version stayed the same
Bundler attempted to update logstash-codec-line but its version stayed the same
Bundler attempted to update logstash-codec-msgpack but its version stayed the same
Bundler attempted to update logstash-codec-multiline but its version stayed the same
Bundler attempted to update logstash-codec-netflow but its version stayed the same
Bundler attempted to update logstash-codec-plain but its version stayed the same
Bundler attempted to update logstash-codec-rubydebug but its version stayed the same
Bundler attempted to update logstash-filter-aggregate but its version stayed the same
Bundler attempted to update logstash-filter-anonymize but its version stayed the same
Bundler attempted to update logstash-filter-cidr but its version stayed the same
Bundler attempted to update logstash-filter-clone but its version stayed the same
Bundler attempted to update logstash-filter-csv but its version stayed the same
Bundler attempted to update logstash-filter-date but its version stayed the same
Bundler attempted to update logstash-filter-de_dot but its version stayed the same
Bundler attempted to update logstash-filter-dissect but its version stayed the same
Bundler attempted to update logstash-filter-dns but its version stayed the same
Bundler attempted to update logstash-filter-drop but its version stayed the same
Bundler attempted to update logstash-filter-fingerprint but its version stayed the same
Bundler attempted to update logstash-filter-geoip but its version stayed the same
Bundler attempted to update logstash-filter-grok but its version stayed the same
Bundler attempted to update logstash-filter-http but its version stayed the same
Bundler attempted to update logstash-filter-json but its version stayed the same
Bundler attempted to update logstash-filter-kv but its version stayed the same
Bundler attempted to update logstash-filter-memcached but its version stayed the same
Bundler attempted to update logstash-filter-metrics but its version stayed the same
Bundler attempted to update logstash-filter-mutate but its version stayed the same
Bundler attempted to update logstash-filter-prune but its version stayed the same
Bundler attempted to update logstash-filter-ruby but its version stayed the same
Bundler attempted to update logstash-filter-sleep but its version stayed the same
Bundler attempted to update logstash-filter-split but its version stayed the same
Bundler attempted to update logstash-filter-syslog_pri but its version stayed the same
Bundler attempted to update logstash-filter-throttle but its version stayed the same
Bundler attempted to update logstash-filter-translate but its version stayed the same
Bundler attempted to update logstash-filter-truncate but its version stayed the same
Bundler attempted to update logstash-filter-urldecode but its version stayed the same
Bundler attempted to update logstash-filter-useragent but its version stayed the same
Bundler attempted to update logstash-filter-uuid but its version stayed the same
Bundler attempted to update logstash-filter-xml but its version stayed the same
Bundler attempted to update logstash-input-beats but its version stayed the same
Bundler attempted to update logstash-input-couchdb_changes but its version stayed the same
Bundler attempted to update logstash-input-dead_letter_queue but its version stayed the same
Bundler attempted to update logstash-input-elastic_serverless_forwarder but its version stayed the same
Bundler attempted to update logstash-input-exec but its version stayed the same
Bundler attempted to update logstash-input-file but its version stayed the same
Bundler attempted to update logstash-input-ganglia but its version stayed the same
Bundler attempted to update logstash-input-gelf but its version stayed the same
Bundler attempted to update logstash-input-generator but its version stayed the same
Bundler attempted to update logstash-input-graphite but its version stayed the same
Bundler attempted to update logstash-input-heartbeat but its version stayed the same
Bundler attempted to update logstash-input-http_poller but its version stayed the same
Bundler attempted to update logstash-input-jms but its version stayed the same
Bundler attempted to update logstash-input-pipe but its version stayed the same
Bundler attempted to update logstash-input-redis but its version stayed the same
Bundler attempted to update logstash-input-stdin but its version stayed the same
Bundler attempted to update logstash-input-syslog but its version stayed the same
Bundler attempted to update logstash-input-tcp but its version stayed the same
Bundler attempted to update logstash-input-twitter but its version stayed the same
Bundler attempted to update logstash-input-udp but its version stayed the same
Bundler attempted to update logstash-input-unix but its version stayed the same
Bundler attempted to update logstash-integration-aws but its version stayed the same
Bundler attempted to update logstash-integration-rabbitmq but its version stayed the same
Bundler attempted to update logstash-output-csv but its version stayed the same
Bundler attempted to update logstash-output-email but its version stayed the same
Bundler attempted to update logstash-output-file but its version stayed the same
Bundler attempted to update logstash-output-graphite but its version stayed the same
Bundler attempted to update logstash-output-http but its version stayed the same
Bundler attempted to update logstash-output-lumberjack but its version stayed the same
Bundler attempted to update logstash-output-nagios but its version stayed the same
Bundler attempted to update logstash-output-null but its version stayed the same
Bundler attempted to update logstash-output-pipe but its version stayed the same
Bundler attempted to update logstash-output-redis but its version stayed the same
Bundler attempted to update logstash-output-stdout but its version stayed the same
Bundler attempted to update logstash-output-tcp but its version stayed the same
Bundler attempted to update logstash-output-udp but its version stayed the same
Bundler attempted to update logstash-output-webhdfs but its version stayed the same
Updated logstash-filter-elastic_integration 0.1.16 to 0.1.17
Updated logstash-filter-elasticsearch 3.16.1 to 3.16.2
Updated logstash-input-azure_event_hubs 1.5.0 to 1.5.1
Updated logstash-input-elasticsearch 4.20.4 to 4.21.1
Updated logstash-input-http 3.9.2 to 3.10.0
Updated logstash-integration-elastic_enterprise_search 3.0.0 to 3.0.1
Updated logstash-integration-jdbc 5.5.0 to 5.5.2
Updated logstash-integration-kafka 11.5.2 to 11.6.0
Updated logstash-integration-logstash 1.0.3 to 1.0.4
Updated logstash-integration-snmp 4.0.4 to 4.0.5
Updated logstash-output-elasticsearch 11.22.9 to 11.22.10

We can see in the second no major versions were upgraded.

Copy link
Member

@jsvd jsvd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've confirmed the behavior works as expected, just missing docs in https://github.com/elastic/logstash/blob/main/docs/static/plugin-manager.asciidoc?plain=1#L102-L113

Copy link
Contributor

📃 DOCS PREVIEWhttps://logstash_bk_16899.docs-preview.app.elstc.co/diff

@yaauie yaauie requested a review from karenzone January 14, 2025 20:11
@@ -112,6 +112,21 @@ bin/logstash-plugin update logstash-input-github <2>
<1> updates all installed plugins
<2> updates only the plugin you specify

[NOTE]
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Preview: https://logstash_bk_16899.docs-preview.app.elstc.co/guide/en/logstash/master/working-with-plugins.html#updating-plugins

I added this as an admonition because in the normal case users shouldn't need to concern themselves with this and will be happy to get the latest fully-compatible version available. It is rare that a user would want to jump across major versions of a plugin because we are taking care to backport non-breaking features and fixes to the previous minor.

@yaauie yaauie requested a review from jsvd January 14, 2025 20:17
Copy link
Member

@jsvd jsvd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doc nitpick, otherwise LGTM

Copy link

Quality Gate passed Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarQube

@elasticmachine
Copy link
Collaborator

💚 Build Succeeded

History

cc @yaauie

Copy link
Contributor

📃 DOCS PREVIEWhttps://logstash_bk_16899.docs-preview.app.elstc.co/diff

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

bin/logstash-plugin update should not include major versions changes by default
3 participants