-1. 设置网卡临时 IP 地址cx5:sudoipaddradd192.168.0.150/24 dev enp3s0f0np0sudoiplinksetenp3s0f0np0 up##sudo ip route replace default via 192.168.1.1 dev enp3s0f0np00. 安装 DOCA安装 CUDA1. 安装 GDRCopy1.1. 编译安装# 下载源代码gitclone https://github.com/NVIDIA/gdrcopy.gitcdgdrcopy/gittaggitcheckout v2.6gitbranch# 编译 UMD 和 KMDmake# UMD 安装在本地查看安装的内容makeinstallprefix/home/ruler/ex_holoscan/tmp09_GDRCopy_gpunetio_open/localcd../local/ tree# UMD 安装到默认系统目录sudomakeinstallsudoldconfig# KMD 安装到系统目录sudomakedrv_instal lsmod|grepgdrlscatinsmod.sh# 将 KMD 插入内核./insmod.sh lsmod|grepgdr# 验证gdrcopy_sanity连个安装命令的打印供参考sudo make installsudo make drv_installrulerruler-Super-Server:~/ex_holoscan/tmp09_GDRCopy_gpunetio_open/gdrcopy$ sudo make install [sudo] password for ruler: awk: cmd. line:1: warning: regexp escape sequence \# is not a known regexp operator awk: cmd. line:1: warning: regexp escape sequence \# is not a known regexp operator awk: cmd. line:1: warning: regexp escape sequence \# is not a known regexp operator awk: cmd. line:1: warning: regexp escape sequence \# is not a known regexp operator cd src \ make LIB_MAJOR_VER2 LIB_MINOR_VER6 make[1]: Entering directory /home/ruler/ex_holoscan/tmp09_GDRCopy_gpunetio_open/gdrcopy/src [INFO] Compiler supports MOVDIR64B GDRAPI_ARCHX86 make[1]: Leaving directory /home/ruler/ex_holoscan/tmp09_GDRCopy_gpunetio_open/gdrcopy/src awk: cmd. line:1: warning: regexp escape sequence \# is not a known regexp operator awk: cmd. line:1: warning: regexp escape sequence \# is not a known regexp operator installing in /usr/local/lib /usr/local/include... src/libgdrapi.so.2.6 - /usr/local/lib/libgdrapi.so.2.6 include/gdrapi.h - /usr/local/include/gdrapi.h include/gdrconfig.h - /usr/local/include/gdrconfig.h cd tests \ make CUDA/usr/local/cuda make[1]: Entering directory /home/ruler/ex_holoscan/tmp09_GDRCopy_gpunetio_open/gdrcopy/tests make[1]: Nothing to be done for all. make[1]: Leaving directory /home/ruler/ex_holoscan/tmp09_GDRCopy_gpunetio_open/gdrcopy/tests cd tests make install DESTBIN/usr/local/bin make[1]: Entering directory /home/ruler/ex_holoscan/tmp09_GDRCopy_gpunetio_open/gdrcopy/tests installing exes in /usr/local/bin... gdrcopy_copybw - /usr/local/bin/gdrcopy_copybw gdrcopy_copylat - /usr/local/bin/gdrcopy_copylat gdrcopy_apiperf - /usr/local/bin/gdrcopy_apiperf gdrcopy_sanity - /usr/local/bin/gdrcopy_sanity gdrcopy_pplat - /usr/local/bin/gdrcopy_pplat cd /usr/local/bin \ ln -sf gdrcopy_copybw copybw \ ln -sf gdrcopy_copylat copylat \ ln -sf gdrcopy_apiperf apiperf \ ln -sf gdrcopy_sanity sanity make[1]: Leaving directory /home/ruler/ex_holoscan/tmp09_GDRCopy_gpunetio_open/gdrcopy/tests rulerruler-Super-Server:~/ex_holoscan/tmp09_GDRCopy_gpunetio_open/gdrcopy$ sudo make drv_install awk: cmd. line:1: warning: regexp escape sequence \# is not a known regexp operator awk: cmd. line:1: warning: regexp escape sequence \# is not a known regexp operator cd src/gdrdrv \ make make[1]: Entering directory /home/ruler/ex_holoscan/tmp09_GDRCopy_gpunetio_open/gdrcopy/src/gdrdrv Picking NVIDIA driver sources from NVIDIA_SRC_DIR/usr/src/nvidia-570.211.01/nvidia. If that does not meet your expectation, you might have a stale driver still around and that might cause problems. Setting NVIDIA_IS_OPENSOURCE Setting HAVE_VM_FLAGS_SETy Setting HAVE_PROC_OPSy make[2]: Entering directory /usr/src/linux-headers-6.8.0-138-generic warning: the compiler differs from the one used to build the kernel The kernel was built by: x86_64-linux-gnu-gcc-12 (Ubuntu 12.3.0-1ubuntu1~22.04.3) 12.3.0 You are using: gcc-12 (Ubuntu 12.3.0-1ubuntu1~22.04.3) 12.3.0 make[2]: Leaving directory /usr/src/linux-headers-6.8.0-138-generic make[1]: Leaving directory /home/ruler/ex_holoscan/tmp09_GDRCopy_gpunetio_open/gdrcopy/src/gdrdrv cd src/gdrdrv \ make install make[1]: Entering directory /home/ruler/ex_holoscan/tmp09_GDRCopy_gpunetio_open/gdrcopy/src/gdrdrv Picking NVIDIA driver sources from NVIDIA_SRC_DIR/usr/src/nvidia-570.211.01/nvidia. If that does not meet your expectation, you might have a stale driver still around and that might cause problems. Setting NVIDIA_IS_OPENSOURCE Setting HAVE_VM_FLAGS_SETy Setting HAVE_PROC_OPSy make[2]: Entering directory /usr/src/linux-headers-6.8.0-138-generic warning: the compiler differs from the one used to build the kernel The kernel was built by: x86_64-linux-gnu-gcc-12 (Ubuntu 12.3.0-1ubuntu1~22.04.3) 12.3.0 You are using: gcc-12 (Ubuntu 12.3.0-1ubuntu1~22.04.3) 12.3.0 make[2]: Leaving directory /usr/src/linux-headers-6.8.0-138-generic [ -d //lib/modules/6.8.0-138-generic//kernel/drivers/misc/ ] || mkdir -p //lib/modules/6.8.0-138-generic//kernel/drivers/misc/ cp gdrdrv.ko //lib/modules/6.8.0-138-generic//kernel/drivers/misc/ if [ ! -n ]; then /sbin/depmod -ae 6.8.0-138-generic; fi depmod: WARNING: -e needs -E or -F make[1]: Leaving directory /home/ruler/ex_holoscan/tmp09_GDRCopy_gpunetio_open/gdrcopy/src/gdrdrv rulerruler-Super-Server:~/ex_holoscan/tmp09_GDRCopy_gpunetio_open/gdrcopy$1.2. gdrcopy_sanity 的工作内容gdrcopy_sanity是 GDRCopyNVIDIA 的 GPUDirect 快速拷贝库自带的功能自检程序源码在 gdrcopy 仓库的tests/sanity.cpp。它的作用是对libgdrapi用户态库 gdrdrv内核模块做一轮完整的功能性验证确认安装和运行环境没问题。它具体测试的内容大致包括打开/初始化gdr_open()能否成功打开/dev/gdrdrv设备这一步就能暴露内核模块没加载的问题。显存映射在 GPU 上分配显存cuMemAlloc然后通过gdr_pin_buffer()gdr_map()把 GPU 显存映射到 CPU 用户空间再gdr_unmap()/gdr_unpin_buffer()解除。测试各种对齐/不对齐的偏移和大小。数据拷贝正确性gdr_copy_to_bar()CPU → GPU 显存写入gdr_copy_from_bar()GPU 显存 → CPU 读取用特定数据模式pattern写入再读回逐字节比对验证拷贝没有数据损坏。CPU 直接读写 BAR通过映射得到的指针直接 memcpy 访问 GPU 显存验证数据一致性。错误处理/边界条件比如重复映射、非法 unmap、未 pin 就 map 等非法调用是否被正确拒绝。多进程 fork 测试映射后 fork 子进程验证句柄和映射在进程间的行为符合预期映射不可跨进程继承等。怎么解读结果全部通过会输出一串PASSED常见失败点gdr_open失败 →gdrdrv内核模块没加载lsmod | grep gdrdrvpin/map 失败 → GPU 不支持 GPUDirect、驱动没开NVreg_EnableGpuFirmware相关设置或 IOMMU 开启导致问题数据校验失败 → 硬件/拓扑问题比如跨 NUMA、P2P 路径异常。简单说它就是 GDRCopy 的冒烟测试跑过全绿说明 GPUDirect 的低延迟 CPU↔GPU 显存拷贝通路是好的。2. 安装 GPUNetIO opensource2.1. 编译安装 GPUNetIO opensourcegitclone https://github.com/NVIDIA-DOCA/gpunetio.gitcdgpunetio/gitbranchgittaggitcheckout v4.0.1gitbranchmake-jlsmakeinstallinstall_examplesPREFIX/home/ruler/ex_holoscan/tmp09_GDRCopy_gpunetio_open/local_gpunetioCUDA_ARCH70cd../local_gpunetio/ treelscdexamples/lsldd gpunetio_verbs_write_latexportLD_LIBRARY_PATH/home/ruler/ex_holoscan/tmp09_GDRCopy_gpunetio_open/local_gpunetio/lib ldd gpunetio_verbs_write_lat ./gpunetio_verbs_write_lat ll ./gpunetio_verbs_put_bw ./gpunetio_verbs_write_bw ./gpunetio_verbs_write_bw--helpcd../..如果显卡是 V100需要制定gpu 架构目前默认是 CUDA_ARCH80cdgpunetiomakecleanmake-jCUDA_ARCH702.2. 运行测试程序2.2.1.step 1: 先启动 服务器端进入等待接收状态$DOCA_GPUNETIO_LOG6./gpunetio_verbs_put_bw-g41:00.0-dmlx5_0step2: 再启动 客户端客户端发送数据给服务器端后会立即退出。DOCA_GPUNETIO_LOG6./gpunetio_verbs_put_bw-g41:00.0-dmlx5_0-c192.168.0.101192.168.0.101 是一个 RoCEv2 网卡 的地址。客户端输出step3: CTLC 关掉 服务器程序会输出验证提示信息Validation successfull! Data received correctly from client如下图2.2.2.2.2.3.3. GDRCopy 与 GPUNetIO 的关系从源码里把整条链路都追完了结论如下。3.1. 先分清 GDRCopy 的两个组成部分GDRCopy 用户态库libgdrapi.so内核模块gdrdrv.ko分工是libgdrapi只是个传声筒把gdr_pin_buffer()/gdr_map()这类请求通过/dev/gdrdrv设备节点转给内核gdrdrv内核模块真正干活的部分——调用 NVIDIA 驱动的 P2P 接口nvidia_p2p_get_pages把 GPU 显存页面 pin 住并建立 CPU 地址空间到 GPU 显存的映射。所以什么时候需要安装内核模块的答案很直接只要有任何程序要执行把 GPU 显存映射给 CPU 访问这个操作gdrdrv就必须已加载insmod gdrdrv.koGDRCopy 源码树里有insmod.sh辅助脚本。模块没加载时/dev/gdrdrv不存在gdr_open()直接失败——libgdrapi 单独存在毫无意义。3.2. 开源 gpunetio 什么时候会用到它源码里的实际行为src/doca_gpunetio.cppdoca_gpunetio_gdrcopy.cpp3.2.1. 启动时探测不强制doca_gpu_create()时执行doca_gpu_gdrcopy_is_supported()dlopen(libgdrapi.so.2)gdr_open()。失败只是记日志GDRCopy usage is disabled库照常工作。还可用环境变量DOCA_GPUNETIO_DISABLE_GDRCOPY主动关掉。3.2.2. 唯一的触发点GPU-CPU 共享内存分配GDRCopy 的 pin/map 只在doca_gpu_mem_alloc()的DOCA_GPU_MEM_TYPE_GPU_CPU及GPU_CPU_DATA_DIRECT分支被调用——cudaMalloc 一块显存同时用 GDRCopy 映射出 CPU 侧指针。而谁会申请这种内存看doca_gpu_verbs_get_qp_dev()的逻辑QP 使用软件模拟 doorbell recordNO_DBR_SW_EMULATED或host 侧折叠 CQ时 → 必须 GPU_CPU 内存这正是CPU-assisted / CPU Proxy 模式示例里-p 1的路径GPU 填 WQE 后CPU 线程要读 GPU 显存里的队列状态和 doorbell record替 GPU 敲网卡门铃——CPU 访问 GPU 显存靠的就是 GDRCopy 映射此路径上有assert(support_gdrcopy)是真依赖启用data-directforce_pcie属性时同理。3.2.3. 主路径完全不用GPU SM 直接敲门铃的模式-p 2也是 AUTO 的优先选择中QP/CQ 用DOCA_GPU_MEM_TYPE_GPU纯显存类型分配doorbell 由 GPU 线程写 UAR 完成——整条 GDAKI 数据面不碰 GDRCopy。另外如果应用自己申请 GPU_CPU 内存而 GDRCopy 不可用代码会优雅降级为CPU 内存 cudaHostRegister性能换可用性。3.3. 结论场景需要gdrdrv内核模块编译 gpunetio不需要GDRCopy 是运行时 dlopenGPU SM 直敲门铃的 GDAKI默认/推荐路径-p 2不需要CPU-assisted 模式-p 1不支持 GPU 敲门铃的老平台需要且是硬依赖应用自己用GPU_CPU/>4. GPU QSFP28 FPGA 延迟测量中的 GPUNetIO 技术核实完了答案是其中两条的组合4.1. ✅ GPUNetIO CPU 控制路径初始化阶段由主机 CPU 用DOCA GPUNetIO 的 verbs CPU APIdoca_verbs_*系列完成打开网卡设备底层仍用ibv_get_device_list枚举但随即通过doca_verbs_bridge_verbs_context_create桥接成 DOCA verbs 上下文用doca_verbs_cq_create/doca_verbs_qp_create创建 CQ 和 QP队列内存用doca_gpu_mem_alloc分配在GPU 显存里最后通过doca_gpu_verbs_export_qp把 QP/CQ 句柄导出给 GPU交给 GPU 内核接管注意 QP 类型是DOCA_VERBS_QP_TYPE_UCUnreliable Connected——正对应论文里说的选择不可靠连接、不做重传。4.2. ✅ RDMA Verbs 双边two-sided的 GPUNetIO GPU 数据路径环回内核forward在 GPU 上的工作方式是接收GPU 线程直接轮询 RQ 的完成队列doca_gpu_dev_verbs_qp_get_cq_rq 解析mlx5_cqe64收到 FPGA 发来的 RoCE SEND 包然后repost_receive补充接收 WQE发送GPU 直接往 SQ 写 WQE、敲门铃doorbell把数据 SEND 回 FPGA收发都需要对端配合SEND/RECV 语义、FPGA 侧 HSB IP 按 RoCE QP 协议发包收包这是典型的双边操作。4.3. ❌ 未使用的四项Verbs CPU 控制路径没有用传统的纯 ibverbs/mlx5dv 手工建队列方式用的是 DOCA verbs 封装单边one-sided没有使用 RDMA Write/Read内核里没有任何 write/read 原语——单边语义里数据去向由发起方指定不适合这种FPGA 推数据、GPU 回发的对等交互Ethernet 数据路径不是裸二层以太网帧收发而是跑在 RoCEIB transport over Ethernet之上DMA 数据路径没有用 DOCA DMA 引擎做显存↔主存拷贝本来就不需要数据全程在显存概括CPU 用 GPUNetIODOCA verbs建链并把队列导出给 GPU之后 GPU 常驻内核以双边 SEND/RECV 方式直接驱动 ConnectX-7完成 RoCE 环回——控制路径和数据路径的分工正好体现了CPU 只初始化、GPU 全权收发的设计。
-1. 设置网卡临时 IP 地址cx5:sudoipaddradd192.168.0.150/24 dev enp3s0f0np0sudoiplinksetenp3s0f0np0 up##sudo ip route replace default via 192.168.1.1 dev enp3s0f0np00. 安装 DOCA安装 CUDA1. 安装 GDRCopy1.1. 编译安装# 下载源代码gitclone https://github.com/NVIDIA/gdrcopy.gitcdgdrcopy/gittaggitcheckout v2.6gitbranch# 编译 UMD 和 KMDmake# UMD 安装在本地查看安装的内容makeinstallprefix/home/ruler/ex_holoscan/tmp09_GDRCopy_gpunetio_open/localcd../local/ tree# UMD 安装到默认系统目录sudomakeinstallsudoldconfig# KMD 安装到系统目录sudomakedrv_instal lsmod|grepgdrlscatinsmod.sh# 将 KMD 插入内核./insmod.sh lsmod|grepgdr# 验证gdrcopy_sanity连个安装命令的打印供参考sudo make installsudo make drv_installrulerruler-Super-Server:~/ex_holoscan/tmp09_GDRCopy_gpunetio_open/gdrcopy$ sudo make install [sudo] password for ruler: awk: cmd. line:1: warning: regexp escape sequence \# is not a known regexp operator awk: cmd. line:1: warning: regexp escape sequence \# is not a known regexp operator awk: cmd. line:1: warning: regexp escape sequence \# is not a known regexp operator awk: cmd. line:1: warning: regexp escape sequence \# is not a known regexp operator cd src \ make LIB_MAJOR_VER2 LIB_MINOR_VER6 make[1]: Entering directory /home/ruler/ex_holoscan/tmp09_GDRCopy_gpunetio_open/gdrcopy/src [INFO] Compiler supports MOVDIR64B GDRAPI_ARCHX86 make[1]: Leaving directory /home/ruler/ex_holoscan/tmp09_GDRCopy_gpunetio_open/gdrcopy/src awk: cmd. line:1: warning: regexp escape sequence \# is not a known regexp operator awk: cmd. line:1: warning: regexp escape sequence \# is not a known regexp operator installing in /usr/local/lib /usr/local/include... src/libgdrapi.so.2.6 - /usr/local/lib/libgdrapi.so.2.6 include/gdrapi.h - /usr/local/include/gdrapi.h include/gdrconfig.h - /usr/local/include/gdrconfig.h cd tests \ make CUDA/usr/local/cuda make[1]: Entering directory /home/ruler/ex_holoscan/tmp09_GDRCopy_gpunetio_open/gdrcopy/tests make[1]: Nothing to be done for all. make[1]: Leaving directory /home/ruler/ex_holoscan/tmp09_GDRCopy_gpunetio_open/gdrcopy/tests cd tests make install DESTBIN/usr/local/bin make[1]: Entering directory /home/ruler/ex_holoscan/tmp09_GDRCopy_gpunetio_open/gdrcopy/tests installing exes in /usr/local/bin... gdrcopy_copybw - /usr/local/bin/gdrcopy_copybw gdrcopy_copylat - /usr/local/bin/gdrcopy_copylat gdrcopy_apiperf - /usr/local/bin/gdrcopy_apiperf gdrcopy_sanity - /usr/local/bin/gdrcopy_sanity gdrcopy_pplat - /usr/local/bin/gdrcopy_pplat cd /usr/local/bin \ ln -sf gdrcopy_copybw copybw \ ln -sf gdrcopy_copylat copylat \ ln -sf gdrcopy_apiperf apiperf \ ln -sf gdrcopy_sanity sanity make[1]: Leaving directory /home/ruler/ex_holoscan/tmp09_GDRCopy_gpunetio_open/gdrcopy/tests rulerruler-Super-Server:~/ex_holoscan/tmp09_GDRCopy_gpunetio_open/gdrcopy$ sudo make drv_install awk: cmd. line:1: warning: regexp escape sequence \# is not a known regexp operator awk: cmd. line:1: warning: regexp escape sequence \# is not a known regexp operator cd src/gdrdrv \ make make[1]: Entering directory /home/ruler/ex_holoscan/tmp09_GDRCopy_gpunetio_open/gdrcopy/src/gdrdrv Picking NVIDIA driver sources from NVIDIA_SRC_DIR/usr/src/nvidia-570.211.01/nvidia. If that does not meet your expectation, you might have a stale driver still around and that might cause problems. Setting NVIDIA_IS_OPENSOURCE Setting HAVE_VM_FLAGS_SETy Setting HAVE_PROC_OPSy make[2]: Entering directory /usr/src/linux-headers-6.8.0-138-generic warning: the compiler differs from the one used to build the kernel The kernel was built by: x86_64-linux-gnu-gcc-12 (Ubuntu 12.3.0-1ubuntu1~22.04.3) 12.3.0 You are using: gcc-12 (Ubuntu 12.3.0-1ubuntu1~22.04.3) 12.3.0 make[2]: Leaving directory /usr/src/linux-headers-6.8.0-138-generic make[1]: Leaving directory /home/ruler/ex_holoscan/tmp09_GDRCopy_gpunetio_open/gdrcopy/src/gdrdrv cd src/gdrdrv \ make install make[1]: Entering directory /home/ruler/ex_holoscan/tmp09_GDRCopy_gpunetio_open/gdrcopy/src/gdrdrv Picking NVIDIA driver sources from NVIDIA_SRC_DIR/usr/src/nvidia-570.211.01/nvidia. If that does not meet your expectation, you might have a stale driver still around and that might cause problems. Setting NVIDIA_IS_OPENSOURCE Setting HAVE_VM_FLAGS_SETy Setting HAVE_PROC_OPSy make[2]: Entering directory /usr/src/linux-headers-6.8.0-138-generic warning: the compiler differs from the one used to build the kernel The kernel was built by: x86_64-linux-gnu-gcc-12 (Ubuntu 12.3.0-1ubuntu1~22.04.3) 12.3.0 You are using: gcc-12 (Ubuntu 12.3.0-1ubuntu1~22.04.3) 12.3.0 make[2]: Leaving directory /usr/src/linux-headers-6.8.0-138-generic [ -d //lib/modules/6.8.0-138-generic//kernel/drivers/misc/ ] || mkdir -p //lib/modules/6.8.0-138-generic//kernel/drivers/misc/ cp gdrdrv.ko //lib/modules/6.8.0-138-generic//kernel/drivers/misc/ if [ ! -n ]; then /sbin/depmod -ae 6.8.0-138-generic; fi depmod: WARNING: -e needs -E or -F make[1]: Leaving directory /home/ruler/ex_holoscan/tmp09_GDRCopy_gpunetio_open/gdrcopy/src/gdrdrv rulerruler-Super-Server:~/ex_holoscan/tmp09_GDRCopy_gpunetio_open/gdrcopy$1.2. gdrcopy_sanity 的工作内容gdrcopy_sanity是 GDRCopyNVIDIA 的 GPUDirect 快速拷贝库自带的功能自检程序源码在 gdrcopy 仓库的tests/sanity.cpp。它的作用是对libgdrapi用户态库 gdrdrv内核模块做一轮完整的功能性验证确认安装和运行环境没问题。它具体测试的内容大致包括打开/初始化gdr_open()能否成功打开/dev/gdrdrv设备这一步就能暴露内核模块没加载的问题。显存映射在 GPU 上分配显存cuMemAlloc然后通过gdr_pin_buffer()gdr_map()把 GPU 显存映射到 CPU 用户空间再gdr_unmap()/gdr_unpin_buffer()解除。测试各种对齐/不对齐的偏移和大小。数据拷贝正确性gdr_copy_to_bar()CPU → GPU 显存写入gdr_copy_from_bar()GPU 显存 → CPU 读取用特定数据模式pattern写入再读回逐字节比对验证拷贝没有数据损坏。CPU 直接读写 BAR通过映射得到的指针直接 memcpy 访问 GPU 显存验证数据一致性。错误处理/边界条件比如重复映射、非法 unmap、未 pin 就 map 等非法调用是否被正确拒绝。多进程 fork 测试映射后 fork 子进程验证句柄和映射在进程间的行为符合预期映射不可跨进程继承等。怎么解读结果全部通过会输出一串PASSED常见失败点gdr_open失败 →gdrdrv内核模块没加载lsmod | grep gdrdrvpin/map 失败 → GPU 不支持 GPUDirect、驱动没开NVreg_EnableGpuFirmware相关设置或 IOMMU 开启导致问题数据校验失败 → 硬件/拓扑问题比如跨 NUMA、P2P 路径异常。简单说它就是 GDRCopy 的冒烟测试跑过全绿说明 GPUDirect 的低延迟 CPU↔GPU 显存拷贝通路是好的。2. 安装 GPUNetIO opensource2.1. 编译安装 GPUNetIO opensourcegitclone https://github.com/NVIDIA-DOCA/gpunetio.gitcdgpunetio/gitbranchgittaggitcheckout v4.0.1gitbranchmake-jlsmakeinstallinstall_examplesPREFIX/home/ruler/ex_holoscan/tmp09_GDRCopy_gpunetio_open/local_gpunetioCUDA_ARCH70cd../local_gpunetio/ treelscdexamples/lsldd gpunetio_verbs_write_latexportLD_LIBRARY_PATH/home/ruler/ex_holoscan/tmp09_GDRCopy_gpunetio_open/local_gpunetio/lib ldd gpunetio_verbs_write_lat ./gpunetio_verbs_write_lat ll ./gpunetio_verbs_put_bw ./gpunetio_verbs_write_bw ./gpunetio_verbs_write_bw--helpcd../..如果显卡是 V100需要制定gpu 架构目前默认是 CUDA_ARCH80cdgpunetiomakecleanmake-jCUDA_ARCH702.2. 运行测试程序2.2.1.step 1: 先启动 服务器端进入等待接收状态$DOCA_GPUNETIO_LOG6./gpunetio_verbs_put_bw-g41:00.0-dmlx5_0step2: 再启动 客户端客户端发送数据给服务器端后会立即退出。DOCA_GPUNETIO_LOG6./gpunetio_verbs_put_bw-g41:00.0-dmlx5_0-c192.168.0.101192.168.0.101 是一个 RoCEv2 网卡 的地址。客户端输出step3: CTLC 关掉 服务器程序会输出验证提示信息Validation successfull! Data received correctly from client如下图2.2.2.2.2.3.3. GDRCopy 与 GPUNetIO 的关系从源码里把整条链路都追完了结论如下。3.1. 先分清 GDRCopy 的两个组成部分GDRCopy 用户态库libgdrapi.so内核模块gdrdrv.ko分工是libgdrapi只是个传声筒把gdr_pin_buffer()/gdr_map()这类请求通过/dev/gdrdrv设备节点转给内核gdrdrv内核模块真正干活的部分——调用 NVIDIA 驱动的 P2P 接口nvidia_p2p_get_pages把 GPU 显存页面 pin 住并建立 CPU 地址空间到 GPU 显存的映射。所以什么时候需要安装内核模块的答案很直接只要有任何程序要执行把 GPU 显存映射给 CPU 访问这个操作gdrdrv就必须已加载insmod gdrdrv.koGDRCopy 源码树里有insmod.sh辅助脚本。模块没加载时/dev/gdrdrv不存在gdr_open()直接失败——libgdrapi 单独存在毫无意义。3.2. 开源 gpunetio 什么时候会用到它源码里的实际行为src/doca_gpunetio.cppdoca_gpunetio_gdrcopy.cpp3.2.1. 启动时探测不强制doca_gpu_create()时执行doca_gpu_gdrcopy_is_supported()dlopen(libgdrapi.so.2)gdr_open()。失败只是记日志GDRCopy usage is disabled库照常工作。还可用环境变量DOCA_GPUNETIO_DISABLE_GDRCOPY主动关掉。3.2.2. 唯一的触发点GPU-CPU 共享内存分配GDRCopy 的 pin/map 只在doca_gpu_mem_alloc()的DOCA_GPU_MEM_TYPE_GPU_CPU及GPU_CPU_DATA_DIRECT分支被调用——cudaMalloc 一块显存同时用 GDRCopy 映射出 CPU 侧指针。而谁会申请这种内存看doca_gpu_verbs_get_qp_dev()的逻辑QP 使用软件模拟 doorbell recordNO_DBR_SW_EMULATED或host 侧折叠 CQ时 → 必须 GPU_CPU 内存这正是CPU-assisted / CPU Proxy 模式示例里-p 1的路径GPU 填 WQE 后CPU 线程要读 GPU 显存里的队列状态和 doorbell record替 GPU 敲网卡门铃——CPU 访问 GPU 显存靠的就是 GDRCopy 映射此路径上有assert(support_gdrcopy)是真依赖启用data-directforce_pcie属性时同理。3.2.3. 主路径完全不用GPU SM 直接敲门铃的模式-p 2也是 AUTO 的优先选择中QP/CQ 用DOCA_GPU_MEM_TYPE_GPU纯显存类型分配doorbell 由 GPU 线程写 UAR 完成——整条 GDAKI 数据面不碰 GDRCopy。另外如果应用自己申请 GPU_CPU 内存而 GDRCopy 不可用代码会优雅降级为CPU 内存 cudaHostRegister性能换可用性。3.3. 结论场景需要gdrdrv内核模块编译 gpunetio不需要GDRCopy 是运行时 dlopenGPU SM 直敲门铃的 GDAKI默认/推荐路径-p 2不需要CPU-assisted 模式-p 1不支持 GPU 敲门铃的老平台需要且是硬依赖应用自己用GPU_CPU/>4. GPU QSFP28 FPGA 延迟测量中的 GPUNetIO 技术核实完了答案是其中两条的组合4.1. ✅ GPUNetIO CPU 控制路径初始化阶段由主机 CPU 用DOCA GPUNetIO 的 verbs CPU APIdoca_verbs_*系列完成打开网卡设备底层仍用ibv_get_device_list枚举但随即通过doca_verbs_bridge_verbs_context_create桥接成 DOCA verbs 上下文用doca_verbs_cq_create/doca_verbs_qp_create创建 CQ 和 QP队列内存用doca_gpu_mem_alloc分配在GPU 显存里最后通过doca_gpu_verbs_export_qp把 QP/CQ 句柄导出给 GPU交给 GPU 内核接管注意 QP 类型是DOCA_VERBS_QP_TYPE_UCUnreliable Connected——正对应论文里说的选择不可靠连接、不做重传。4.2. ✅ RDMA Verbs 双边two-sided的 GPUNetIO GPU 数据路径环回内核forward在 GPU 上的工作方式是接收GPU 线程直接轮询 RQ 的完成队列doca_gpu_dev_verbs_qp_get_cq_rq 解析mlx5_cqe64收到 FPGA 发来的 RoCE SEND 包然后repost_receive补充接收 WQE发送GPU 直接往 SQ 写 WQE、敲门铃doorbell把数据 SEND 回 FPGA收发都需要对端配合SEND/RECV 语义、FPGA 侧 HSB IP 按 RoCE QP 协议发包收包这是典型的双边操作。4.3. ❌ 未使用的四项Verbs CPU 控制路径没有用传统的纯 ibverbs/mlx5dv 手工建队列方式用的是 DOCA verbs 封装单边one-sided没有使用 RDMA Write/Read内核里没有任何 write/read 原语——单边语义里数据去向由发起方指定不适合这种FPGA 推数据、GPU 回发的对等交互Ethernet 数据路径不是裸二层以太网帧收发而是跑在 RoCEIB transport over Ethernet之上DMA 数据路径没有用 DOCA DMA 引擎做显存↔主存拷贝本来就不需要数据全程在显存概括CPU 用 GPUNetIODOCA verbs建链并把队列导出给 GPU之后 GPU 常驻内核以双边 SEND/RECV 方式直接驱动 ConnectX-7完成 RoCE 环回——控制路径和数据路径的分工正好体现了CPU 只初始化、GPU 全权收发的设计。
