Code

Idle source keran#93

Code

Draft

KeranYang wants to merge 59 commits intomainKeranYang/numaflow:mainfrom idle-source-keranKeranYang/numaflow:idle-source-keranCopy head branch name to clipboard

+2,390-1,127

Commits

Commits on Nov 23, 2023

• initial commit

  Show description for 5bd354c
  

  yhl25

  authored and

  chandankumar4

  committed
  5bd354c

  Copy full SHA for 5bd354c

• Publish idle watermark

  Show description for aeff817
  chandankumar4

  committed
  aeff817

  Copy full SHA for aeff817

• Add idle watermark publish methods

  Show description for 3bedd7e
  chandankumar4

  committed
  3bedd7e

  Copy full SHA for 3bedd7e

• update idle watermark methods

  Show description for 188b6e3
  chandankumar4

  committed
  188b6e3

  Copy full SHA for 188b6e3

• calculate the idle watermark based on configs

  Show description for 5cff192
  chandankumar4

  committed
  5cff192

  Copy full SHA for 5cff192

• Fix unit test cases

  Show description for e8dec81
  chandankumar4

  committed
  e8dec81

  Copy full SHA for e8dec81

• Fix comments

  Show description for 112bc3b
  chandankumar4

  committed
  112bc3b

  Copy full SHA for 112bc3b

Commits on Nov 24, 2023

• Add sourcePublisher interface

  Show description for f835af0
  chandankumar4

  committed
  f835af0

  Copy full SHA for f835af0

• Update PublishIdleWatermarks for kafka

  Show description for 6c44c90
  chandankumar4

  committed
  6c44c90

  Copy full SHA for 6c44c90

Commits on Nov 25, 2023

• Fix lint check

  Show description for e2e4ca5
  chandankumar4

  committed
  e2e4ca5

  Copy full SHA for e2e4ca5

Commits on Nov 26, 2023

• Add unit for idle watermark

  Show description for bdd532e
  chandankumar4

  committed
  bdd532e

  Copy full SHA for bdd532e

Commits on Nov 27, 2023

• Resolve review comment

  Show description for 9525c3f
  chandankumar4

  committed
  9525c3f

  Copy full SHA for 9525c3f

• Fix codegen

  Show description for 8b05029
  chandankumar4

  committed
  8b05029

  Copy full SHA for 8b05029

• Add e2e test for idle source

  Show description for 1796132
  chandankumar4

  committed
  1796132

  Copy full SHA for 1796132

Commits on Nov 28, 2023

• Update currentWatermark status

  Show description for 4a47521
  chandankumar4

  committed
  4a47521

  Copy full SHA for 4a47521

• minor changes

  Show description for 175fa32
  yhl25

  committed
  175fa32

  Copy full SHA for 175fa32

• Fix test and lint

  Show description for f0e9a2b
  chandankumar4

  committed
  f0e9a2b

  Copy full SHA for f0e9a2b

• Add maxDelay in idle source config

  Show description for 785167b
  chandankumar4

  committed
  785167b

  Copy full SHA for 785167b

Commits on Nov 29, 2023

• chore: rename source

  Show description for 7896573
  vigith

  committed
  7896573

  Copy full SHA for 7896573

• Update interface and minor review comment

  Show description for c6bc8e9
  chandankumar4

  committed
  c6bc8e9

  Copy full SHA for c6bc8e9

• Update the comments

  Show description for 7c4c8f9
  chandankumar4

  committed
  7c4c8f9

  Copy full SHA for 7c4c8f9

• chore: rename idler

  Show description for d8a908f
  vigith

  committed
  d8a908f

  Copy full SHA for d8a908f

Commits on Nov 30, 2023

• refactor watermark and sources

  Show description for 1c975b4
  yhl25

  committed
  1c975b4

  Copy full SHA for 1c975b4

• Merge branch 'main' of github.com:numaproj/numaflow into idle-source

  yhl25

  committed
  d47409c

  Copy full SHA for d47409c

• resolving conflicts

  Show description for cf0bab9
  yhl25

  committed
  cf0bab9

  Copy full SHA for cf0bab9

• Fix code comments

  Show description for b33d860
  chandankumar4

  committed
  b33d860

  Copy full SHA for b33d860

• update codegen

  Show description for a157c87
  chandankumar4

  committed
  a157c87

  Copy full SHA for a157c87

• header

  Show description for 7f79bd4
  vigith

  committed
  7f79bd4

  Copy full SHA for 7f79bd4

• chore: code review

  Show description for f2e68ba
  vigith

  committed
  f2e68ba

  Copy full SHA for f2e68ba

• Add comments and invert condition for fn stepIntervalHasPassed

  Show description for a81c6dd
  chandankumar4

  committed
  a81c6dd

  Copy full SHA for a81c6dd

Commits on Dec 4, 2023

• fix idle source

  Show description for 2bf72df
  yhl25

  committed
  2bf72df

  Copy full SHA for 2bf72df

• Merge branch 'idle-source' of github.com:numaproj/numaflow into idle-source

  yhl25

  committed
  18edcbe

  Copy full SHA for 18edcbe

• resolve conflicts

  Show description for c0228ed
  yhl25

  committed
  c0228ed

  Copy full SHA for c0228ed

• publish idle wm to all the toVertexBuffers if src is idling

  Show description for 14c6947
  yhl25

  committed
  14c6947

  Copy full SHA for 14c6947

• introducing SourceReader with support for getting partitions

  Show description for 5586bdb
  yhl25

  committed
  5586bdb

  Copy full SHA for 5586bdb

• fix index out of range

  Show description for 760706f
  yhl25

  committed
  760706f

  Copy full SHA for 760706f

• codegen

  Show description for cd08b62
  yhl25

  committed
  cd08b62

  Copy full SHA for cd08b62

• Merge branch 'main' into idle-source

  chandankumar4

  authored
  a36c514

  Copy full SHA for a36c514

• Add validation for idle source config

  Show description for 89e2b08
  chandankumar4

  committed
  89e2b08

  Copy full SHA for 89e2b08

• more changes

  Show description for 50ac14b
  yhl25

  committed
  50ac14b

  Copy full SHA for 50ac14b

• Update comment and idle source e2e test

  Show description for c8fd06f
  chandankumar4

  committed
  c8fd06f

  Copy full SHA for c8fd06f

Commits on Dec 5, 2023

• Update pipeline validation and comments

  Show description for 45f4f11
  chandankumar4

  committed
  45f4f11

  Copy full SHA for 45f4f11

• avoid adding delay for idle watermark

  Show description for 81c8d3d
  yhl25

  committed
  81c8d3d

  Copy full SHA for 81c8d3d

• remove noack for source

  Show description for 7aad126
  yhl25

  committed
  7aad126

  Copy full SHA for 7aad126

• Merge branch 'main' of github.com:numaproj/numaflow into idle-source

  yhl25

  committed
  b5936bd

  Copy full SHA for b5936bd

• review comments

  Show description for dc92984
  yhl25

  committed
  dc92984

  Copy full SHA for dc92984

• Update e2e test

  Show description for 3fa8e96
  chandankumar4

  committed
  3fa8e96

  Copy full SHA for 3fa8e96

• increase sink timeout

  Show description for 1023b5d
  yhl25

  committed
  1023b5d

  Copy full SHA for 1023b5d

• Update idleSource config

  Show description for 64b2e4b
  chandankumar4

  committed
  64b2e4b

  Copy full SHA for 64b2e4b

• chore: code review

  Show description for ee2010f
  vigith

  committed
  ee2010f

  Copy full SHA for ee2010f

• update map

  Show description for 6d951a7
  yhl25

  committed
  6d951a7

  Copy full SHA for 6d951a7

• Merge branch 'idle-source' of github.com:numaproj/numaflow into idle-source

  yhl25

  committed
  63af898

  Copy full SHA for 63af898

Commits on Dec 6, 2023

• chore: code review

  Show description for ff12653
  vigith

  committed
  ff12653

  Copy full SHA for ff12653

• return early

  Show description for a4b0823
  yhl25

  committed
  a4b0823

  Copy full SHA for a4b0823

• add validation and unit tests

  Show description for ca2cd8d
  chandankumar4

  committed
  ca2cd8d

  Copy full SHA for ca2cd8d

• review

  Show description for 025fa4d
  KeranYang

  committed
  025fa4d

  Copy full SHA for 025fa4d

• without idling

  Show description for 00f385a
  KeranYang

  committed
  00f385a

  Copy full SHA for 00f385a

• Revert "without idling"

  Show description for fc4bfac
  KeranYang

  committed
  fc4bfac

  Copy full SHA for fc4bfac

• .

  Show description for 2d2e632
  KeranYang

  committed
  2d2e632

  Copy full SHA for 2d2e632