190 lines
5.3 KiB
Markdown
190 lines
5.3 KiB
Markdown
## deepspeed docker image build
|
|
|
|
```shell
|
|
docker-compose -f docker-compose_pytorch1.13.yml build
|
|
docker-compose -f docker-compose_pytorch2.3.yml build
|
|
```
|
|
|
|
## 镜像测试命令
|
|
|
|
```shell
|
|
nvidia-smi
|
|
nvcc -V
|
|
ninja --version
|
|
ds_report
|
|
python -c "import torch; print('torch:', torch.__version__, torch)"
|
|
python -c "import torch; print('CUDA available:', torch.cuda.is_available())"
|
|
python -c "import deepspeed; deepspeed.ops.op_builder.CPUAdamBuilder().load()"
|
|
python -c "from flash_attn import flash_attn_func, flash_attn_varlen_func"
|
|
python -c "import apex.amp; print('Apex is installed and the amp module is available.')"
|
|
python -c "from xformers import ops as xops"
|
|
ibstat
|
|
ofed_info -s
|
|
mst version
|
|
mpirun --version
|
|
```
|
|
|
|
## 物理机更新内核
|
|
|
|
```shell
|
|
uname -r # 5.4.0-144-generic
|
|
lsb_release -a
|
|
sudo apt-get update # This will update the repositories list
|
|
sudo apt-get upgrade # This will update all the necessary packages on your system
|
|
sudo apt-get dist-upgrade # This will add/remove any needed packages
|
|
reboot # You may need this since sometimes after a upgrade/dist-upgrade, there are some left over entries that get fixed after a reboot
|
|
sudo apt-get install linux-headers-$(uname -r) # This should work now
|
|
```
|
|
|
|
## test command
|
|
|
|
```shell
|
|
docker run -it --gpus all --name deepspeed_test --shm-size=1gb --rm hotwa/deepspeed:latest /bin/bash
|
|
```
|
|
|
|
## [查询GPU 架构 给变量赋值](https://blog.csdn.net/zong596568821xp/article/details/106411024)
|
|
|
|
```shell
|
|
git clone https://github.com/NVIDIA-AI-IOT/deepstream_tlt_apps.git
|
|
cd deepstream_tlt_apps/TRT-OSS/x86
|
|
nvcc deviceQuery.cpp -o deviceQuery
|
|
./deviceQuery
|
|
```
|
|
|
|
H100 输出
|
|
|
|
```shell
|
|
(base) root@node19:~/bgpt/deepstream_tlt_apps/TRT-OSS/x86# ./deviceQuery
|
|
Detected 8 CUDA Capable device(s)
|
|
|
|
Device 0: "NVIDIA H100 80GB HBM3"
|
|
CUDA Driver Version / Runtime Version 12.4 / 10.1
|
|
CUDA Capability Major/Minor version number: 9.0
|
|
|
|
Device 1: "NVIDIA H100 80GB HBM3"
|
|
CUDA Driver Version / Runtime Version 12.4 / 10.1
|
|
CUDA Capability Major/Minor version number: 9.0
|
|
|
|
Device 2: "NVIDIA H100 80GB HBM3"
|
|
CUDA Driver Version / Runtime Version 12.4 / 10.1
|
|
CUDA Capability Major/Minor version number: 9.0
|
|
|
|
Device 3: "NVIDIA H100 80GB HBM3"
|
|
CUDA Driver Version / Runtime Version 12.4 / 10.1
|
|
CUDA Capability Major/Minor version number: 9.0
|
|
|
|
Device 4: "NVIDIA H100 80GB HBM3"
|
|
CUDA Driver Version / Runtime Version 12.4 / 10.1
|
|
CUDA Capability Major/Minor version number: 9.0
|
|
|
|
Device 5: "NVIDIA H100 80GB HBM3"
|
|
CUDA Driver Version / Runtime Version 12.4 / 10.1
|
|
CUDA Capability Major/Minor version number: 9.0
|
|
|
|
Device 6: "NVIDIA H100 80GB HBM3"
|
|
CUDA Driver Version / Runtime Version 12.4 / 10.1
|
|
CUDA Capability Major/Minor version number: 9.0
|
|
|
|
Device 7: "NVIDIA H100 80GB HBM3"
|
|
CUDA Driver Version / Runtime Version 12.4 / 10.1
|
|
CUDA Capability Major/Minor version number: 9.0
|
|
|
|
```
|
|
|
|
|
|
## DeepSpeed hostfile 分发
|
|
|
|
要手动分发 hostfile 并进行分布式安装，你需要以下几个步骤：
|
|
|
|
1. 准备 hostfile
|
|
确保 hostfile 文件包含所有参与的主机及其配置。
|
|
|
|
示例 hostfile 内容：
|
|
|
|
```plaintext
|
|
host1 slots=4
|
|
host2 slots=4
|
|
host3 slots=8
|
|
```
|
|
|
|
2. 确保 SSH 配置正确
|
|
确保你能够通过 SSH 无密码登录到所有主机。可以使用 ssh-keygen 和 ssh-copy-id 配置 SSH 密钥。
|
|
|
|
生成 SSH 密钥（如果尚未生成）：
|
|
|
|
```shell
|
|
ssh-keygen -t rsa
|
|
```
|
|
|
|
将 SSH 公钥复制到每个主机：
|
|
|
|
```shell
|
|
ssh-copy-id user@host1
|
|
ssh-copy-id user@host2
|
|
ssh-copy-id user@host3
|
|
```
|
|
|
|
3. 创建临时目录并复制 wheel 文件
|
|
在所有主机上创建一个临时目录，用于存放分发的 wheel 文件。
|
|
|
|
```shell
|
|
export PDSH_RCMD_TYPE=ssh
|
|
hosts=$(cat /path/to/your/hostfile | awk '{print $1}' | paste -sd ",")
|
|
tmp_wheel_path="/tmp/deepspeed_wheels"
|
|
|
|
pdsh -w $hosts "mkdir -pv ${tmp_wheel_path}"
|
|
pdcp -w $hosts dist/deepspeed*.whl ${tmp_wheel_path}/
|
|
pdcp -w $hosts requirements/requirements.txt ${tmp_wheel_path}/
|
|
```
|
|
|
|
4. 在每个主机上安装 DeepSpeed 和依赖项
|
|
在所有主机上安装 DeepSpeed 和所需的依赖项。
|
|
|
|
```shell
|
|
pdsh -w $hosts "pip install ${tmp_wheel_path}/deepspeed*.whl"
|
|
pdsh -w $hosts "pip install -r ${tmp_wheel_path}/requirements.txt"
|
|
```
|
|
|
|
5. 清理临时文件
|
|
安装完成后，删除所有主机上的临时文件。
|
|
|
|
```shell
|
|
pdsh -w $hosts "rm -rf ${tmp_wheel_path}"
|
|
```
|
|
|
|
详细步骤
|
|
确保 SSH 配置正确：
|
|
|
|
```shell
|
|
ssh-keygen -t rsa
|
|
ssh-copy-id user@host1
|
|
ssh-copy-id user@host2
|
|
ssh-copy-id user@host3
|
|
```
|
|
|
|
创建临时目录并复制文件：
|
|
|
|
```shell
|
|
export PDSH_RCMD_TYPE=ssh
|
|
hosts=$(cat /path/to/your/hostfile | awk '{print $1}' | paste -sd ",")
|
|
tmp_wheel_path="/tmp/deepspeed_wheels"
|
|
|
|
pdsh -w $hosts "mkdir -pv ${tmp_wheel_path}"
|
|
pdcp -w $hosts dist/deepspeed*.whl ${tmp_wheel_path}/
|
|
pdcp -w $hosts requirements/requirements.txt ${tmp_wheel_path}/
|
|
```
|
|
|
|
在所有主机上安装 DeepSpeed 和依赖项：
|
|
|
|
```shell
|
|
pdsh -w $hosts "pip install ${tmp_wheel_path}/deepspeed*.whl"
|
|
pdsh -w $hosts "pip install -r ${tmp_wheel_path}/requirements.txt"
|
|
```
|
|
|
|
清理临时文件：
|
|
|
|
```shell
|
|
pdsh -w $hosts "rm -rf ${tmp_wheel_path}"
|
|
```
|
|
|
|
通过这些步骤，你可以手动分发 hostfile 并在多个主机上安装 DeepSpeed 和其依赖项。这种方法确保了每个主机的环境配置一致，从而支持分布式训练或部署。 |