Code

Megatron-LM style Sequence Parallel#1257

Code

Merged

Quentin-Anthony merged 28 commits intomainEleutherAI/gpt-neox:mainfrom 812-megatron-seq-parallelEleutherAI/gpt-neox:812-megatron-seq-parallelCopy head branch name to clipboardAug 23, 2024

+349-33

Commits

Commits on Jul 30, 2024

• first draft (shape errors occurring)

  haileyschoelkopf

  committed
  ad4f0a4

  Copy full SHA for ad4f0a4

Commits on Jul 31, 2024

• training works (but poor convergence)

  haileyschoelkopf

  committed
  dc4c99b

  Copy full SHA for dc4c99b

Commits on Aug 1, 2024

• debugging progress: current commit works if we do regular TP via impl-ing AR in rowparallel as RS then AG

  haileyschoelkopf

  committed
  3ccd3ba

  Copy full SHA for 3ccd3ba

Commits on Aug 8, 2024

• Update NeoXArgs docs automatically

  

  github-actions

  committed
  73aa0aa

  Copy full SHA for 73aa0aa

• push most recent code (updated mark_norms fn, back to 'real' sequence parallel)

  haileyschoelkopf

  committed
  92ed0cc

  Copy full SHA for 92ed0cc

• Merge branch '812-megatron-seq-parallel' of https://github.com/EleutherAI/gpt-neox into 812-megatron-seq-parallel

  haileyschoelkopf

  committed
  c93c1b4

  Copy full SHA for c93c1b4

• Update NeoXArgs docs automatically

  

  github-actions

  committed
  9c1e7b9

  Copy full SHA for 9c1e7b9

Commits on Aug 13, 2024

• Fix LayerNorm all reduce gradient hook

  bclyang

  committed
  651e24e

  Copy full SHA for 651e24e

Commits on Aug 14, 2024

• Sum instead of average for LayerNorm gradient all reduce

  bclyang

  committed
  9a43318

  Copy full SHA for 9a43318

• Update NeoXArgs docs automatically

  

  github-actions

  committed
  c0561d6

  Copy full SHA for c0561d6

• Merge pull request #1259 from EleutherAI/fix-ln-hooks

  Show description for 9945910
  haileyschoelkopf

  authored
  9945910

  Copy full SHA for 9945910

• Update NeoXArgs docs automatically

  

  github-actions

  committed
  2c5dc5a

  Copy full SHA for 2c5dc5a

Commits on Aug 17, 2024

• Fix gather and reduce scatter ops on sequence dimension

  bclyang

  committed
  9d883de

  Copy full SHA for 9d883de

Commits on Aug 18, 2024

• Fix sequence parallel with tied weight embeddings

  bclyang

  committed
  28a5a62

  Copy full SHA for 28a5a62

• Update NeoXArgs docs automatically

  

  github-actions

  committed
  5427d9d

  Copy full SHA for 5427d9d

Commits on Aug 19, 2024

• Merge pull request #1260 from EleutherAI/fix-seq-dim-reducegatherdactter

  Show description for b0d9398
  haileyschoelkopf

  authored
  b0d9398

  Copy full SHA for b0d9398

• cleanup pass + add MoE arguments.py guard

  haileyschoelkopf

  committed
  8f26029

  Copy full SHA for 8f26029

• pre-commit and clean up comments

  Quentin-Anthony

  committed
  d9db749

  Copy full SHA for d9db749

• remove vestigial debug code

  Quentin-Anthony

  committed
  aafbbce

  Copy full SHA for aafbbce

• remove unused debugging code

  haileyschoelkopf

  committed
  ba682e7

  Copy full SHA for ba682e7

• remove dummy test config

  haileyschoelkopf

  committed
  8455de7

  Copy full SHA for 8455de7

• Merge branch '812-megatron-seq-parallel' of https://github.com/EleutherAI/gpt-neox into 812-megatron-seq-parallel

  haileyschoelkopf

  committed
  9ce982e

  Copy full SHA for 9ce982e

• update fp32_allreduce to handle fp16 ; don't cast to fp32 for gathers

  haileyschoelkopf

  committed
  ab11a6a

  Copy full SHA for ab11a6a

• run linter on the rest of the files

  haileyschoelkopf

  committed
  f26b886

  Copy full SHA for f26b886

Commits on Aug 22, 2024

• Improve performance of sequence parallel gather, scatter, and reduce

  bclyang

  committed
  8e7400f

  Copy full SHA for 8e7400f

• Add comment

  bclyang

  committed
  53d0ae8

  Copy full SHA for 53d0ae8

• Update NeoXArgs docs automatically

  

  github-actions

  committed
  05f5cec

  Copy full SHA for 05f5cec

• Merge pull request #1263 from EleutherAI/improve-seq-parallel-perf

  Show description for 1661db6
  haileyschoelkopf

  authored
  1661db6

  Copy full SHA for 1661db6