Support all the softmax extensions and cherry-pick transformer-related commits#101

Open

hubertlu-tw wants to merge 25 commits intomasterfrom

dev/hubertlu/run_transformer

Commits on Dec 28, 2022

Add a generic fused softmax (#1440)

yidong72
authored and
hubertlu-tw
committed
[transformer] unittest: less mem consumption for generic softmax (#1448)

crcrpar
authored and
hubertlu-tw
committed
Use `xmlrunner.XMLTestRunner` accordingly in `tests/L0/run_test.py` (#1451)

crcrpar
authored and
hubertlu-tw
committed
Update mlp_cuda test (#1425)

crcrpar
authored and
hubertlu-tw
committed
Use `xmlrunner.XMLTestRunner` accordingly in `tests/L0/run_test.py` (#1451)

crcrpar
authored and
hubertlu-tw
committed
Update `apex.mlp` to use fp16 in `autocast` (#1477)

crcrpar
authored and
hubertlu-tw
committed
Skip flaky test for ROCm
hubertlu-tw
committed
Label smoothing in vocab parallel cross entropy (#1457)

MaximumEntropy
authored and
hubertlu-tw
committed
introducing `APEX_RUN_WITH_SLOW_TESTS` env var (#1489)

crcrpar
authored and
hubertlu-tw
committed

Commits on Dec 29, 2022

[contrib][DistributedFusedAdam] Support overlapped grad sync with Megatron pipeline parallelism (#1475)

timmoon10
authored and
hubertlu-tw
committed
[transformer] Allow for skipping stream sync (#1505)

authored and
hubertlu-tw
committed
Skip test_grad_scaler in test_dist_adam.py
hubertlu-tw
committed
Add run_rocm_extensions.sh for skipFlakyTest and skipIfRocm
hubertlu-tw
committed
add tearDown (#1508)

Aidyn-A
authored and
hubertlu-tw
committed
Support overlapped grad sync with Megatron interleaved pipeline parallelism (#1514)

timmoon10
authored and
hubertlu-tw
committed
update run_transformer tests to default to using pytorch native UCC instead of torch_ucc (#1495)

Fuzzkatt
authored and
hubertlu-tw
committed
update exception message (#1524)

crcrpar
authored and
hubertlu-tw
committed
Update megatron fused softmax follow megatron-lm (#1539)

yaoyu-33
authored and
hubertlu-tw
committed
Resolve filename collision issue in compilation on ROCm (Ref: #77)
hubertlu-tw
committed
Refactor run_transformer bert minimal and gpt minimal tests (#1540)

Fuzzkatt
authored and
hubertlu-tw
committed
check `is_ucc_available` (#1523)

crcrpar
authored and
hubertlu-tw
committed

Commits on Dec 30, 2022

Support fused_weight_gradient_mlp_cuda for ROCm
hubertlu-tw
committed
Fix bugs in run_rocm_extensions.py
hubertlu-tw
committed
Move test_label_smoothing.py to xentropy folder
hubertlu-tw
committed
Add some test folders for those ROCm does not support
hubertlu-tw
committed