Code

Modernize MosaicBERT#440

Code

Open

Skylion007 wants to merge 19 commits intomosaicml:mainmosaicml/examples:mainfrom Skylion007:skylion007/add-fa2-to-bertSkylion007/mosaicml-examples:skylion007/add-fa2-to-bertCopy head branch name to clipboard

+648-474

Commits

Commits on Jan 2, 2024

• Modernize MosaicBERT

  Skylion007

  committed
  b809a7b

  Copy full SHA for b809a7b

• Reviewer suggestions & import refactor

  Skylion007

  committed
  dbe8d64

  Copy full SHA for dbe8d64

Commits on Jan 4, 2024

• Fix broken tests

  Skylion007

  committed
  6a8d85b

  Copy full SHA for 6a8d85b

• Cast to bfloat16 if possible

  Skylion007

  committed
  4cbdec1

  Copy full SHA for 4cbdec1

• Update streaming kwarg

  Skylion007

  committed
  6ecb798

  Copy full SHA for 6ecb798

• Improve HALF_DTYPE selection

  Skylion007

  committed
  eb93f3b

  Copy full SHA for eb93f3b

• revert back to old HALF selection

  Skylion007

  committed
  58d9ffe

  Copy full SHA for 58d9ffe

Commits on Jan 5, 2024

• update documentation

  jacobfulano

  committed
  c6d58e4

  Copy full SHA for c6d58e4

• clean up documentation

  jacobfulano

  committed
  d5ec672

  Copy full SHA for d5ec672

• update yamls to mosaicml/pytorch:2.1.1_cu121-python3.10-ubuntu20.04

  jacobfulano

  committed
  bcd10d2

  Copy full SHA for bcd10d2

• clean up comments and documentation

  jacobfulano

  committed
  e9e1213

  Copy full SHA for e9e1213

• linting

  jacobfulano

  committed
  ea82b1b

  Copy full SHA for ea82b1b

• small formatting for pre-commit hook

  jacobfulano

  committed
  26313a5

  Copy full SHA for 26313a5

• linting and type: ignore

  jacobfulano

  committed
  329ed51

  Copy full SHA for 329ed51

• linting

  jacobfulano

  committed
  95a4753

  Copy full SHA for 95a4753

• type ignore

  jacobfulano

  committed
  28ccb4f

  Copy full SHA for 28ccb4f

• yapf precommit hook

  jacobfulano

  committed
  2cdb6af

  Copy full SHA for 2cdb6af

• remove bug for self.slopes

  jacobfulano

  committed
  cb78d90

  Copy full SHA for cb78d90

Commits on Jan 8, 2024

• remove fused_layernorm with apex dependency and include low_precision_layernorm

  jacobfulano

  committed
  c44effa

  Copy full SHA for c44effa