Sample Usage Guide【免费下载链接】geGEGraph Engine是面向昇腾的图编译器和执行器提供了计算图优化、多流并行、内存复用和模型下沉等技术手段加速模型执行效率减少模型内存占用。 GE 提供对 PyTorch、TensorFlow 前端的友好接入能力并同时支持 onnx、pb 等主流模型格式的解析与编译。项目地址: https://gitcode.com/cann/geFeature DescriptionThis sample demonstrates capture tensor functionality using MatMulAdd fusion to GEMM pass, providing two verification methods: online inference and ATC offline model compilation. The pass uses eager style API and fusion interface.Directory Structure├── src │ ├──fuse_matmul_add_pass.cpp // pass implementation file ├── CMakeLists.txt // build script ├── data | ├──torch_gen_onnx.py // torch script for exporting onnx | ├──torch_forward_1.py // torch script for online inference, pass executed successfully | ├──torch_forward_2.py // torch script for online inference, pass blocked |—— gen_es_api | |——CMakeLists.txt // build script for generating eager style apiEnvironment RequirementsCompiler: GCC 7.3.xPython and dependencies: python3.9, pytorch2.1Environment preparation completed.Implementation StepsDefine classFuseMatMulAndAddPassinheriting fromPatternFusionPass.Override three functions from base classPatternFusionPass:Patternsdefines matching templates for identifying topologies matching the template in the graph.pattern-CaptureTensor()captures tensor, tensor structure:{NodeIo,index}.MeetRequirementsfilters topologies matched by template.match_result-GetCapturedTensor(kAddCaptureIdx,add_node);reads captured NodeIo for checking.Replacementdefines replacement part.match_result-GetCapturedTensor(kMatMulCaptureIdx, matmul_node);reads captured NodeIo to extract attribute values.RegisterFuseMatMulAndAddPassas custom fusion pass with execution phase BeforeInferShape.Program CompilationConfigure environment variables.Run environment setup script from software package:source ${ASCEND_PATH}/set_env.sh${ASCEND_PATH}is cann path under CANN software package installation directory. Replace with actual installation path, e.g.,${INSTALL_PATH}/cann.ModifyCMakeLists.txtas needed.ASCEND_PATH: Default software package path. If$ASCEND_HOME_PATHset via set_env.sh, no modification needed.target_include_directories: Required header files. For this sample, no modification needed. For custom development, add header files below the example without deleting existing items. If network has custom operators, add custom operator prototype definition headers.target_link_libraries: Required libraries. For this sample, no modification needed. For custom development, add libraries below the example without deleting existing items.Do not link other SOs from software package to avoid compatibility issues during future upgrades.Execute sequentially:mkdir build cd build cmake ..Run make to compile custom pass so, then install dynamic library libfuse_matmul_add_for_capture_tensor_sample_pass.so to custom fusion pass directory via make install. Optional parameter-j$(nproc)can be added after make for parallel build tasks,$(nproc)dynamically gets CPU core count.make -j$(nproc) fuse_matmul_add_for_capture_tensor_sample_pass make installAfter sample verification, run the following command to clean custom pass so installed under CANN package to avoid affecting subsequent UT/ST:make clean_custom_passProgram ExecutionConfigure environment variables (if already done, skip).Run environment setup script:source ${ASCEND_PATH}/set_env.shReplace${ASCEND_PATH}with actual software package installation path.Use ATC offline inference.Set environment variable to dump model graph during compilation:export DUMP_GE_GRAPH1Enter data directory and execute .py file to export onnx (uses torch onnx exporter, depends on additional Python package onnx, ensure installed before running. ATC tool currently supports onnx opset_version up to 18, if torch exports higher version by default, specify explicitly, see script comments):python torch_gen_onnx.pyAfter execution, .onnx format model file named model.onnx generated in data directory.Execute ATC tool command (for detailed ATC tool instructions, visit Ascend Documentation and search for ATC Offline Model Compilation Tool), modifysoc_versionbased on actual environment:atc --model./model.onnx --framework5 --soc_versionxxx --output./modelLog shows:Define pattern for FuseMatMulAndAddPass in capture tensor sample Define MeetRequirements for FuseMatMulAndAddPass in capture tensor sample Define replacement for FuseMatMulAndAddPass in capture tensor sampleOnline inferenceSet environment variable to dump model graph during compilation:export DUMP_GE_GRAPH1Enter data directory and execute .py file for online inference (ensure torch_npu plugin installed for online inference), executetorch_forward_1.py:python torch_forward_1.pyFor torch_forward_1.py, log shows:Define pattern for FuseMatMulAndAddPass in capture tensor sample Define MeetRequirements for FuseMatMulAndAddPass in capture tensor sample Define replacement for FuseMatMulAndAddPass in capture tensor sampleExecutetorch_forward_2.py:python torch_forward_2.pyFor torch_forward_2.py, log shows:Define pattern for FuseMatMulAndAddPass in capture tensor sample Define MeetRequirements for FuseMatMulAndAddPass in capture tensor sample Only support Add inputs are fp32View resultsAfter execution, series of .pbtxt files generated in directory. Compare the following dump graphs:ge_onnx_xxxxx_PreRunBegin.pbtxtdump graph before executionge_onnx_xxxxx_RunCustomPassBeforeInferShape.pbtxtcustom pass dump graph before InferShape executionModel optimized as expected, i.e., MatMul and Add replaced by GEMM.If results not as expected, set following environment variables (for atc command, also add parameter--logdebug) to print logs to screen for troubleshooting.export ASCEND_SLOG_PRINT_TO_STDOUT1 #print logs to screen export ASCEND_GLOBAL_LOG_LEVEL0 #log level debug【免费下载链接】geGEGraph Engine是面向昇腾的图编译器和执行器提供了计算图优化、多流并行、内存复用和模型下沉等技术手段加速模型执行效率减少模型内存占用。 GE 提供对 PyTorch、TensorFlow 前端的友好接入能力并同时支持 onnx、pb 等主流模型格式的解析与编译。项目地址: https://gitcode.com/cann/ge创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考
Sample Usage Guide【免费下载链接】geGEGraph Engine是面向昇腾的图编译器和执行器提供了计算图优化、多流并行、内存复用和模型下沉等技术手段加速模型执行效率减少模型内存占用。 GE 提供对 PyTorch、TensorFlow 前端的友好接入能力并同时支持 onnx、pb 等主流模型格式的解析与编译。项目地址: https://gitcode.com/cann/geFeature DescriptionThis sample demonstrates capture tensor functionality using MatMulAdd fusion to GEMM pass, providing two verification methods: online inference and ATC offline model compilation. The pass uses eager style API and fusion interface.Directory Structure├── src │ ├──fuse_matmul_add_pass.cpp // pass implementation file ├── CMakeLists.txt // build script ├── data | ├──torch_gen_onnx.py // torch script for exporting onnx | ├──torch_forward_1.py // torch script for online inference, pass executed successfully | ├──torch_forward_2.py // torch script for online inference, pass blocked |—— gen_es_api | |——CMakeLists.txt // build script for generating eager style apiEnvironment RequirementsCompiler: GCC 7.3.xPython and dependencies: python3.9, pytorch2.1Environment preparation completed.Implementation StepsDefine classFuseMatMulAndAddPassinheriting fromPatternFusionPass.Override three functions from base classPatternFusionPass:Patternsdefines matching templates for identifying topologies matching the template in the graph.pattern-CaptureTensor()captures tensor, tensor structure:{NodeIo,index}.MeetRequirementsfilters topologies matched by template.match_result-GetCapturedTensor(kAddCaptureIdx,add_node);reads captured NodeIo for checking.Replacementdefines replacement part.match_result-GetCapturedTensor(kMatMulCaptureIdx, matmul_node);reads captured NodeIo to extract attribute values.RegisterFuseMatMulAndAddPassas custom fusion pass with execution phase BeforeInferShape.Program CompilationConfigure environment variables.Run environment setup script from software package:source ${ASCEND_PATH}/set_env.sh${ASCEND_PATH}is cann path under CANN software package installation directory. Replace with actual installation path, e.g.,${INSTALL_PATH}/cann.ModifyCMakeLists.txtas needed.ASCEND_PATH: Default software package path. If$ASCEND_HOME_PATHset via set_env.sh, no modification needed.target_include_directories: Required header files. For this sample, no modification needed. For custom development, add header files below the example without deleting existing items. If network has custom operators, add custom operator prototype definition headers.target_link_libraries: Required libraries. For this sample, no modification needed. For custom development, add libraries below the example without deleting existing items.Do not link other SOs from software package to avoid compatibility issues during future upgrades.Execute sequentially:mkdir build cd build cmake ..Run make to compile custom pass so, then install dynamic library libfuse_matmul_add_for_capture_tensor_sample_pass.so to custom fusion pass directory via make install. Optional parameter-j$(nproc)can be added after make for parallel build tasks,$(nproc)dynamically gets CPU core count.make -j$(nproc) fuse_matmul_add_for_capture_tensor_sample_pass make installAfter sample verification, run the following command to clean custom pass so installed under CANN package to avoid affecting subsequent UT/ST:make clean_custom_passProgram ExecutionConfigure environment variables (if already done, skip).Run environment setup script:source ${ASCEND_PATH}/set_env.shReplace${ASCEND_PATH}with actual software package installation path.Use ATC offline inference.Set environment variable to dump model graph during compilation:export DUMP_GE_GRAPH1Enter data directory and execute .py file to export onnx (uses torch onnx exporter, depends on additional Python package onnx, ensure installed before running. ATC tool currently supports onnx opset_version up to 18, if torch exports higher version by default, specify explicitly, see script comments):python torch_gen_onnx.pyAfter execution, .onnx format model file named model.onnx generated in data directory.Execute ATC tool command (for detailed ATC tool instructions, visit Ascend Documentation and search for ATC Offline Model Compilation Tool), modifysoc_versionbased on actual environment:atc --model./model.onnx --framework5 --soc_versionxxx --output./modelLog shows:Define pattern for FuseMatMulAndAddPass in capture tensor sample Define MeetRequirements for FuseMatMulAndAddPass in capture tensor sample Define replacement for FuseMatMulAndAddPass in capture tensor sampleOnline inferenceSet environment variable to dump model graph during compilation:export DUMP_GE_GRAPH1Enter data directory and execute .py file for online inference (ensure torch_npu plugin installed for online inference), executetorch_forward_1.py:python torch_forward_1.pyFor torch_forward_1.py, log shows:Define pattern for FuseMatMulAndAddPass in capture tensor sample Define MeetRequirements for FuseMatMulAndAddPass in capture tensor sample Define replacement for FuseMatMulAndAddPass in capture tensor sampleExecutetorch_forward_2.py:python torch_forward_2.pyFor torch_forward_2.py, log shows:Define pattern for FuseMatMulAndAddPass in capture tensor sample Define MeetRequirements for FuseMatMulAndAddPass in capture tensor sample Only support Add inputs are fp32View resultsAfter execution, series of .pbtxt files generated in directory. Compare the following dump graphs:ge_onnx_xxxxx_PreRunBegin.pbtxtdump graph before executionge_onnx_xxxxx_RunCustomPassBeforeInferShape.pbtxtcustom pass dump graph before InferShape executionModel optimized as expected, i.e., MatMul and Add replaced by GEMM.If results not as expected, set following environment variables (for atc command, also add parameter--logdebug) to print logs to screen for troubleshooting.export ASCEND_SLOG_PRINT_TO_STDOUT1 #print logs to screen export ASCEND_GLOBAL_LOG_LEVEL0 #log level debug【免费下载链接】geGEGraph Engine是面向昇腾的图编译器和执行器提供了计算图优化、多流并行、内存复用和模型下沉等技术手段加速模型执行效率减少模型内存占用。 GE 提供对 PyTorch、TensorFlow 前端的友好接入能力并同时支持 onnx、pb 等主流模型格式的解析与编译。项目地址: https://gitcode.com/cann/ge创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考
