How much would work it be to add support for normal pytorch checkpointing?

[RuABraun]

Distributed checkpointing while just doing DDP is very buggy for me.

If it's not a big change (I notice there is an option DTensor=False which makes me hopeful), where would one start?

[tsunghsienlee]

Hi @RuABraun ,

Checkpointing is probably our weakest part because the lack of real usages. I remember @runame mentioned to me before on similar issue.

Unfortunately that we might not have time to fix this but I am willing to listen to you if you have any idea on make it work with PyTorch checkpinting.

[tsunghsienlee]

Hi @RuABraun ,

To give you an update, we have @wz337 and @jialun-zhang looking into this. Likely we will have a standard PyTorch checkpoint solution soon.