Code

Bnb/training with obs#252

Code

Draft

bnb32 wants to merge 32 commits intomainNREL/sup3r:mainfrom bnb/training_with_obsNREL/sup3r:bnb/training_with_obsCopy head branch name to clipboard

+2,627-1,708

Commits

Commits on Jan 4, 2025

• Refactor parallelization of bias calculations

  Show description for aff87b6
  bnb32

  committed
  aff87b6

  Copy full SHA for aff87b6

• dual sampler, queue, and batch handler with obs. modifying Sup3rDataset to work with three data members.

  bnb32

  committed
  e726f12

  Copy full SHA for e726f12

• training with obs test

  bnb32

  committed
  4548a18

  Copy full SHA for 4548a18

• split up interface and abstact model

  bnb32

  committed
  d2fe03d

  Copy full SHA for d2fe03d

• made dual batch queue flexible enough to account for additional obs member in data

  bnb32

  committed
  9dc0143

  Copy full SHA for 9dc0143

• tensorboard mixin moved to model utilities. dual queue completely absorbed WithObs variant.

  bnb32

  committed
  7e79caa

  Copy full SHA for 7e79caa

• integrated dual sampler with obs into base dual sampler.

  bnb32

  committed
  6191b28

  Copy full SHA for 6191b28

• examples added to DataHandler doc string. Some instructions on sup3rwind model training added to examples README.

  bnb32

  committed
  4fdff8c

  Copy full SHA for 4fdff8c

• removed namedtuple from Sup3rDataset to make Sup3rDataset picklable.

  bnb32

  committed
  e836057

  Copy full SHA for e836057

• parallel batch queue test added.

  bnb32

  committed
  1e0fb29

  Copy full SHA for 1e0fb29

• namedtuple -> DsetTuple missing attr fix

  bnb32

  committed
  8c15444

  Copy full SHA for 8c15444

• gust added to era download variables. len dunder added to Container class to enable passing xr.Dataset and like objects directly to BatchHandlers without needing to invoke .data

  bnb32

  committed
  373cb68

  Copy full SHA for 373cb68

• computing before reshaping is 2x faster.

  bnb32

  committed
  4c8d77f

  Copy full SHA for 4c8d77f

• obs_index fix - sampler needs to use hr_out_features for the obs member.

  bnb32

  committed
  e1d1ac5

  Copy full SHA for e1d1ac5

• split up calc_loss and calc_loss_obs

  bnb32

  committed
  536beb1

  Copy full SHA for 536beb1

• Optional run_qa flag in DualRasterizer. Queue shape fix for queues with obs data

  bnb32

  committed
  196becf

  Copy full SHA for 196becf

• run_qa=True default for DualRasterizer

  bnb32

  committed
  2a99d94

  Copy full SHA for 2a99d94

• better tracking of batch counting. (this can be tricky for parallel queueing, since batches can be sampled directly if there are none in the queue).

  bnb32

  committed
  c0d4d0a

  Copy full SHA for c0d4d0a

• missed compute call for slow batching. this was hidden by queueing and dequeueing since this would cast to tensors.

  bnb32

  committed
  586130d

  Copy full SHA for 586130d

• Included convert to tensor in sample_batch. Test for training with max_workers > 1.

  bnb32

  committed
  1e29b68

  Copy full SHA for 1e29b68

• cc batch handler test fix

  bnb32

  committed
  fde57b6

  Copy full SHA for fde57b6

• added test for new disc with "valid" padding

  bnb32

  committed
  9b718d7

  Copy full SHA for 9b718d7

• parallel sampling batch sampling test.

  bnb32

  committed
  ed51133

  Copy full SHA for ed51133

• removed workers tests. max_workers > 1 still not consistently faster. just sampling is, except for macos, but training is not.

  bnb32

  committed
  39a28bb

  Copy full SHA for 39a28bb

• Sup3rGanWithObs model subclass. Other misc model refactoring.

  bnb32

  committed
  0c3eb44

  Copy full SHA for 0c3eb44

• bias test fixes

  bnb32

  committed
  7b62d65

  Copy full SHA for 7b62d65

Commits on Jan 5, 2025

• additional bias refact: _run base method and _get_run_kwargs method.

  bnb32

  committed
  a8eea08

  Copy full SHA for a8eea08

• moved _run method to bias correction interface AbstractBiasCorrection

  bnb32

  committed
  ee8e237

  Copy full SHA for ee8e237

Commits on Jan 6, 2025

• moved _run method to bias correction interface AbstractBiasCorrection

  bnb32

  committed
  2d3d4a9

  Copy full SHA for 2d3d4a9

Commits on Jan 8, 2025

• interpolation adjustment: option to include single levels in interpolation along with isobars. Doing so can sometimes lead to issues if values computed from pressure levels differ from the single l…

  Show description for c939301
  bnb32

  committed
  c939301

  Copy full SHA for c939301

• fix: tensorboard issue with loss obs details

  bnb32

  committed
  ab75b77

  Copy full SHA for ab75b77

• test fixes

  bnb32

  committed
  99cbac5

  Copy full SHA for 99cbac5