Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for "uuid" type in tracking_column_type #170

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 18 additions & 4 deletions docs/input-jdbc.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -127,7 +127,7 @@ Here is the list:

|==========================================================
|sql_last_value | The value used to calculate which rows to query. Before any query is run,
this is set to Thursday, 1 January 1970, or 0 if `use_column_value` is true and
this is set to Thursday, 1 January 1970, 0, or 00000000-0000-0000-0000-000000000000 if `use_column_value` is true and
`tracking_column` is set. It is updated accordingly after subsequent queries are run.
|offset, size| Values used with manual paging mode to explicitly implement the paging.
Supported only if <<plugins-{type}s-{plugin}-jdbc_paging_enabled>> is enabled and
Expand Down Expand Up @@ -240,7 +240,7 @@ This plugin supports the following configuration options plus the <<plugins-{typ
| <<plugins-{type}s-{plugin}-statement_filepath>> |a valid filesystem path|No
| <<plugins-{type}s-{plugin}-target>> | {logstash-ref}/field-references-deepdive.html[field reference] | No
| <<plugins-{type}s-{plugin}-tracking_column>> |<<string,string>>|No
| <<plugins-{type}s-{plugin}-tracking_column_type>> |<<string,string>>, one of `["numeric", "timestamp"]`|No
| <<plugins-{type}s-{plugin}-tracking_column_type>> |<<string,string>>, one of `["numeric", "timestamp", "uuid"]`|No
| <<plugins-{type}s-{plugin}-use_column_value>> |<<boolean,boolean>>|No
| <<plugins-{type}s-{plugin}-use_prepared_statements>> |<<boolean,boolean>>|No
|=======================================================================
Expand Down Expand Up @@ -645,10 +645,24 @@ The column whose value is to be tracked if `use_column_value` is set to `true`
[id="plugins-{type}s-{plugin}-tracking_column_type"]
===== `tracking_column_type`

* Value can be any of: `numeric`, `timestamp`
* Value can be any of: `numeric`, `timestamp`, `uuid`
* Default value is `"numeric"`

Type of tracking column. Currently only "numeric" and "timestamp"
Type of tracking column. Currently only "numeric", "timestamp" and "uuid"

NOTE: Example of uuid type usage in PostgreSQL.
[source,ruby]
---------------------------------------------------------------------------------------------------
input {
jdbc {
statement => "SELECT id, mycolumn1, mycolumn2 FROM my_table WHERE id > :sql_last_value::uuid"
use_column_value => true
tracking_column => "id"
tracking_column_type => "uuid"
# ... other configuration bits
}
}
---------------------------------------------------------------------------------------------------

[id="plugins-{type}s-{plugin}-use_column_value"]
===== `use_column_value`
Expand Down
4 changes: 2 additions & 2 deletions lib/logstash/inputs/jdbc.rb
Original file line number Diff line number Diff line change
Expand Up @@ -191,8 +191,8 @@ module LogStash module Inputs class Jdbc < LogStash::Inputs::Base
# If tracking column value rather than timestamp, the column whose value is to be tracked
config :tracking_column, :validate => :string

# Type of tracking column. Currently only "numeric" and "timestamp"
config :tracking_column_type, :validate => ['numeric', 'timestamp'], :default => 'numeric'
# Type of tracking column. Currently only "numeric", "timestamp" and "uuid"
config :tracking_column_type, :validate => ['numeric', 'timestamp', 'uuid'], :default => 'numeric'

# Whether the previous run state should be preserved
config :clean_run, :validate => :boolean, :default => false
Expand Down
25 changes: 21 additions & 4 deletions lib/logstash/plugin_mixins/jdbc/value_tracking.rb
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,15 @@ def self.build_last_value_tracker(plugin)
handler.clean
end

if plugin.use_column_value && plugin.tracking_column_type == "numeric"
# use this irrespective of the jdbc_default_timezone setting
NumericValueTracker.new(handler)
tracking_column_types = ['numeric', 'uuid']
if plugin.use_column_value && tracking_column_types.include?(plugin.tracking_column_type)
case plugin.tracking_column_type
when "numeric"
# use this irrespective of the jdbc_default_timezone setting
NumericValueTracker.new(handler)
when "uuid"
UuidValueTracker.new(handler)
end
else
if plugin.jdbc_default_timezone.nil?
# no TZ stuff for Sequel, use Time
Expand All @@ -34,7 +40,7 @@ def initialize(handler)
end

if Psych::VERSION&.split('.')&.first.to_i >= 4
YAML_PERMITTED_CLASSES = [::DateTime, ::Time, ::BigDecimal].freeze
YAML_PERMITTED_CLASSES = [::DateTime, ::Time, ::BigDecimal, ::String].freeze
def self.load_yaml(source)
Psych::safe_load(source, permitted_classes: YAML_PERMITTED_CLASSES)
end
Expand Down Expand Up @@ -81,6 +87,17 @@ def set_value(value)
end
end

class UuidValueTracker < ValueTracking
def set_initial
common_set_initial(:uuid, "00000000-0000-0000-0000-000000000000")
end

def set_value(value)
return unless value.is_a?(String)
@value = value
end
end

class DateTimeValueTracker < ValueTracking
def set_initial
common_set_initial(:to_datetime, DateTime.new(1970))
Expand Down
8 changes: 5 additions & 3 deletions spec/inputs/jdbc_spec.rb
Original file line number Diff line number Diff line change
Expand Up @@ -35,11 +35,12 @@
db.create_table :test_table do
DateTime :created_at
BigDecimal :big_num
String :uuid
Integer :num
String :string
DateTime :custom_time
end
db << "CREATE TABLE types_table (num INTEGER, string VARCHAR(255), started_at DATE, custom_time TIMESTAMP, ranking DECIMAL(16,6))"
db << "CREATE TABLE types_table (num INTEGER, string VARCHAR(255), started_at DATE, custom_time TIMESTAMP, ranking DECIMAL(16,6), uuid VARCHAR(36))"
db << "CREATE TABLE test1_table (num INTEGER, string VARCHAR(255), custom_time TIMESTAMP, created_at TIMESTAMP)"
end
end
Expand Down Expand Up @@ -1522,7 +1523,7 @@
end

before do
db << "INSERT INTO types_table (num, string, started_at, custom_time, ranking) VALUES (1, 'A test', '1999-12-31', '1999-12-31 23:59:59', 95.67)"
db << "INSERT INTO types_table (num, string, started_at, custom_time, ranking, uuid) VALUES (1, 'A test', '1999-12-31', '1999-12-31 23:59:59', 95.67, '018f15f3-6cfd-7a1b-b70f-d97ed8a73128')"

plugin.register
end
Expand All @@ -1539,6 +1540,7 @@
expect(event.get("started_at")).to be_a_logstash_timestamp_equivalent_to("1999-12-31T00:00:00.000Z")
expect(event.get("custom_time")).to be_a_logstash_timestamp_equivalent_to("1999-12-31T23:59:59.000Z")
expect(event.get("ranking").to_f).to eq(95.67)
expect(event.get("uuid")).to eq("018f15f3-6cfd-7a1b-b70f-d97ed8a73128")
end
end

Expand All @@ -1552,7 +1554,7 @@
end

before(:each) do
db << "INSERT INTO types_table (num, string, started_at, custom_time, ranking) VALUES (1, 'A test', '1999-12-31', '2021-11-07 01:23:45', 95.67)"
db << "INSERT INTO types_table (num, string, started_at, custom_time, ranking, uuid) VALUES (1, 'A test', '1999-12-31', '2021-11-07 01:23:45', 95.67, '018f15f3-6cfd-7a1b-b70f-d97ed8a73128')"
plugin.register
end

Expand Down