Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test tensorflow efa #4449

Merged
merged 24 commits into from
Dec 5, 2024
+10 −1
Merged

test tensorflow efa #4449

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
fa53cb3
test efa
Nov 27, 2024
aed12ce
enable efa test
Nov 27, 2024
222ac3b
run efa test
Nov 27, 2024
913cb63
export cuda home
Nov 27, 2024
823b000
load lib
Nov 27, 2024
e0a1112
add cuda home to mpi run
Nov 28, 2024
aa54242
check cuda home when build tests
Nov 28, 2024
9eaa8fe
change cuda home
Dec 2, 2024
6c10a3f
remove cuda home from mpi
Dec 2, 2024
ed43c9c
nccl_ofi_tuner
Dec 2, 2024
dbacfd3
remove # -x NCCL_TUNER_PLUGIN=/usr/local/lib/libnccl-ofi-tuner.so
Dec 2, 2024
fb199d9
use product image
Dec 2, 2024
ccfafc1
add back tuner
Dec 2, 2024
65de5b0
remove tuner
Dec 3, 2024
ef12dd7
enable all ec2 tests
Dec 3, 2024
369f743
test pytorch efa
Dec 3, 2024
ca6a4ed
pin pynvml version
Dec 4, 2024
e2456f9
use tag override
Dec 4, 2024
715b097
use current image test
Dec 4, 2024
9f6c8cc
add -y to uninstall pynvml
Dec 4, 2024
b3600a0
revert toml
Dec 5, 2024
9ffad4c
Merge branch 'master' into tf-216-tr
Yadan-Wei Dec 5, 2024
df286f2
add back PR Context and pin pynvml version
Dec 5, 2024
f378e30
modify comments
Dec 5, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions tensorflow/training/buildspec-2-16-ec2.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@ images:
tag: !join [ *VERSION, "-", *DEVICE_TYPE, "-", *TAG_PYTHON_VERSION, "-", *OS_VERSION, "-ec2" ]
latest_release_tag: !join [ *VERSION, "-", *DEVICE_TYPE, "-", *TAG_PYTHON_VERSION, "-", *OS_VERSION,
"-ec2" ]
# build_tag_override: "beta:2.16.2-cpu-py310-ubuntu20.04-ec2"
docker_file: !join [ docker/, *SHORT_VERSION, /, *DOCKER_PYTHON_VERSION, /Dockerfile., *DEVICE_TYPE ]
target: ec2
enable_test_promotion: true
Expand All @@ -55,6 +56,7 @@ images:
tag: !join [ *VERSION, "-", *DEVICE_TYPE, "-", *TAG_PYTHON_VERSION, "-", *CUDA_VERSION, "-", *OS_VERSION, "-ec2" ]
latest_release_tag: !join [ *VERSION, "-", *DEVICE_TYPE, "-", *TAG_PYTHON_VERSION, "-", *CUDA_VERSION, "-",
*OS_VERSION, "-ec2" ]
# build_tag_override: "beta:2.16.2-gpu-py310-cu123-ubuntu20.04-ec2"
docker_file: !join [ docker/, *SHORT_VERSION, /, *DOCKER_PYTHON_VERSION, /, *CUDA_VERSION, /Dockerfile.,
*DEVICE_TYPE ]
target: ec2
Expand Down
1 change: 0 additions & 1 deletion test/dlc_tests/container_tests/bin/efa/testEFA
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,6 @@ check_efa_nccl_all_reduce(){
# the container. Not using full-paths of mpirun and other executables because these paths can change across PyTorch
# versions in DLC images.
mpirun -x FI_PROVIDER="efa" -n $NODES -N $GPU_COUNT --hostfile $NUM_HOSTS_FILE \
-x NCCL_TUNER_PLUGIN=/usr/local/lib/libnccl-ofi-tuner.so \
-x NCCL_DEBUG=INFO ${USE_DEVICE_RDMA_ARG} -x NCCL_PROTO=simple -x NCCL_ALGO=ring -x RDMAV_FORK_SAFE=1 \
-x PATH -x LD_LIBRARY_PATH=${CUDA_HOME}/lib:${CUDA_HOME}/lib64:$LD_LIBRARY_PATH \
-x NCCL_SOCKET_IFNAME=^lo --mca pml ^cm --mca btl tcp,self --mca btl_tcp_if_exclude lo,docker0 --bind-to none \
Expand Down
8 changes: 8 additions & 0 deletions test/dlc_tests/container_tests/bin/pytorch_tests/setupPyTorchBackendTest
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,4 +31,12 @@ git clone https://github.com/pytorch/benchmark.git
cd benchmark
git checkout 350bb04 # pin commit for Python 3.11 support
pip install -r requirements.txt

# pin pynvml to avoid issues with 12.0.0
# https://github.com/pytorch/benchmark/issues/2552
# PR: https://github.com/pytorch/benchmark/pull/2553
# when this issue is fixed can revert back
pip uninstall pynvml -y
pip install "pynvml<12"

pip install numba
Loading