You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When using s3fs, I saw this concerning message in the logs:
Future exception was never retrieved
future: <Future finished exception=ConnectionError('Connection lost')>
aiohttp.client_exceptions.ClientOSError: [Errno 32] Broken pipe
The above exception was the direct cause of the following exception:
ConnectionError: Connection lost
Luckily, I was using s3fs to output model checkpoints for training a deep learning model, and the next checkpoint succesfully saved, however, we should really probably properly handle this exception by either forwarding it to the caller, or alternatively, engaging in a low level retry to restore the connection and resume the upload.
The text was updated successfully, but these errors were encountered:
Did this actually involve a failed connection or transfer, or is it something that happens for example during interpreter shutdown? As far I know, all connections should be awaited, although I can imagine that in some cases a timeout (with associated error) might happen first.
When using s3fs, I saw this concerning message in the logs:
Luckily, I was using s3fs to output model checkpoints for training a deep learning model, and the next checkpoint succesfully saved, however, we should really probably properly handle this exception by either forwarding it to the caller, or alternatively, engaging in a low level retry to restore the connection and resume the upload.
The text was updated successfully, but these errors were encountered: