Questions about the loss used for optimizing the proxy model

[clarkkent0618]

@sangmichaelxie It seems that the loss used for optimizing the proxy model in the code is different from the one described in the paper.

loss = (pertoken_loss * curr_domain_weights.detach()).sum() / normalizer

In the code, you directly use the proxy model's own loss here to optimize. But in the paper, the loss seems to be the minimax loss which uses the excess loss. So which one should I conform? Or there is something wrong with my understanding. Thanks.

[yuzc19]

I have another question. When training the main model, what is the difference between resampling the data from the new distribution or using new weights to re-weight loss? Will these two have a significant performance gap?

[sangmichaelxie]

In the code, you directly use the proxy model's own loss here to optimize. But in the paper, the loss seems to be the minimax loss which uses the excess loss. So which one should I conform? Or there is something wrong with my understanding. Thanks.

The reference model loss is a constant with respect to the proxy model's parameters, so it doesn't affect the proxy model update and we omit it. The reference model loss does affect the domain weight update.

When training the main model, what is the difference between resampling the data from the new distribution or using new weights to re-weight loss? Will these two have a significant performance gap?

Check out the main paragraph in pg 9 and Table 3b in the paper.

[yuzc19]

I checked it, and it makes sense. Thank you!