测试DuckDB LuaJit插件通过ffi调用c语言动态库性能

关键步骤有2步。1.用c编译器(gcc/tcc等)编译动态库tcc -shared -fPIC -o libsudoku_c.so sudoku_solve.c2.在DuckDB中把动态库中的函数注册为luajit udfSELECT * FROM luajit_module(mode : compile, sql_name : sudoku_c, source : return function(p) local ffi require(ffi) ffi.cdef[[ int sudoku_solve(const char *p, char *out); ]] local lib ffi.load(/mnt/c/d/libsudoku_c.so) local buf ffi.new(char[82]) local rc lib.sudoku_solve(p, buf) if rc ~ 1 then return NULL end return ffi.string(buf) end );然后用luajit_s等函数调用udf。在Windows中调用不成功。Microsoft Windows [版本 10.0.26200.8655] (c) Microsoft Corporation。保留所有权利。 C:\Users\ltcd\d C:\dduckdb155 DuckDB v1.5.5 (Variegata) Enter .help for usage hints. memory D load luajit; memory D C:\dtcc\tcc -shared -fPIC -o libsudoku_c.so sudoku_solve.c C:\dduckdb155 DuckDB v1.5.5 (Variegata) Enter .help for usage hints. memory D load luajit; memory D SELECT * FROM luajit_module(mode : compile, sql_name : sudoku_c, source : return function(p) local ffi require(ffi) ffi.cdef[[ int sudoku_solve(const char *p, char *out); ]] local lib ffi.load(/d/libsudoku_c.so) local buf ffi.new(char[82]) local rc lib.sudoku_solve(p, buf) if rc ~ 1 then return NULL end return ffi.string(buf) end ); ┌─────────┬─────────┬─────────┬─────────┬─────────┬──────────┐ │ ok │ mode │ phase │ message │ detail │ sql_name │ │ boolean │ varchar │ varchar │ varchar │ varchar │ varchar │ ├─────────┼─────────┼─────────┼─────────┼─────────┼──────────┤ │ true │ compile │ compile │ OK │ │ sudoku_c │ └─────────┴─────────┴─────────┴─────────┴─────────┴──────────┘ memory D SELECT luajit_s(sudoku_c, 000001002000020030004500600007600050080090006100005800001004000070900003400030020); ┌───────────────────────────────────────────────────────────────────────────────────────────────────────────┐ │ luajit_s(sudoku_c, 000001002000020030004500600007600050080090006100005800001004000070900003400030020) │ │ varchar │ ├───────────────────────────────────────────────────────────────────────────────────────────────────────────┤ │ NULL │ └───────────────────────────────────────────────────────────────────────────────────────────────────────────┘在WSL Linux中调用成功。首先用apt install 安装Tcc。C:\dwsl rootDESKTOP-59T6U68:/mnt/c/d# tcc Command tcc not found, but can be installed with: apt install tcc rootDESKTOP-59T6U68:/mnt/c/d# apt install tcc Reading package lists... Done Building dependency tree... Done Reading state information... Done The following NEW packages will be installed: tcc 0 upgraded, 1 newly installed, 0 to remove and 48 not upgraded. Need to get 278 kB of archives. After this operation, 824 kB of additional disk space will be used. Get:1 http://archive.ubuntu.com/ubuntu jammy/universe amd64 tcc amd64 0.9.27git20200814.62c30a4a-1 [278 kB] Fetched 278 kB in 1s (205 kB/s) Selecting previously unselected package tcc. (Reading database ... 84088 files and directories currently installed.) Preparing to unpack .../tcc_0.9.27git20200814.62c30a4a-1_amd64.deb ... Unpacking tcc (0.9.27git20200814.62c30a4a-1) ... Setting up tcc (0.9.27git20200814.62c30a4a-1) ... Processing triggers for install-info (6.8-4build1) ... Processing triggers for man-db (2.10.2-1) ... rootDESKTOP-59T6U68:/mnt/c/d# tcc -shared -fPIC -o libsudoku_c2.so sudoku_solve.c rootDESKTOP-59T6U68:/mnt/c/d# ./duckdb155 DuckDB v1.5.5 (Variegata) Enter .help for usage hints. memory D .timer on memory D load luajit; Run Time (s): real 0.015 user 0.004387 sys 0.000000 memory D SELECT * FROM luajit_module(mode : compile, sql_name : sudoku_c, source : return function(p) local ffi require(ffi) ffi.cdef[[ int sudoku_solve(const char *p, char *out); ]] local lib ffi.load(/mnt/c/d/libsudoku_c2.so) local buf ffi.new(char[82]) local rc lib.sudoku_solve(p, buf) if rc ~ 1 then return NULL end return ffi.string(buf) end ); ┌─────────┬─────────┬─────────┬─────────┬─────────┬──────────┐ │ ok │ mode │ phase │ message │ detail │ sql_name │ │ boolean │ varchar │ varchar │ varchar │ varchar │ varchar │ ├─────────┼─────────┼─────────┼─────────┼─────────┼──────────┤ │ true │ compile │ compile │ OK │ │ sudoku_c │ └─────────┴─────────┴─────────┴─────────┴─────────┴──────────┘ Run Time (s): real 0.008 user 0.002350 sys 0.000048 memory D SELECT max(luajit_s(sudoku_c, column0)) from (from read_csv(sudoku17.txt,header0) limit 100); ┌───────────────────────────────────────────────────────────────────────────────────┐ │ max(luajit_s(sudoku_c, column0)) │ │ varchar │ ├───────────────────────────────────────────────────────────────────────────────────┤ │ 978654213645231798213879654796518432521347986384926175437182569159463827862795341 │ └───────────────────────────────────────────────────────────────────────────────────┘ Run Time (s): real 0.950 user 0.900484 sys 0.008561 memory D rootDESKTOP-59T6U68:/mnt/c/d# tcc -shared -fPIC -o libsudoku_c3.so dlxdll.c rootDESKTOP-59T6U68:/mnt/c/d# exit exit memory D SELECT * FROM luajit_module(mode : compile, sql_name : sudoku_c, source : return function(p) local ffi require(ffi) ffi.cdef[[ int sudoku_solve(const char *p, char *out); ]] local lib ffi.load(/mnt/c/d/libsudoku_c3.so) local buf ffi.new(char[82]) local rc lib.sudoku_solve(p, buf) if rc ~ 1 then return NULL end return ffi.string(buf) end ); ┌─────────┬─────────┬─────────┬─────────┬─────────┬──────────┐ │ ok │ mode │ phase │ message │ detail │ sql_name │ │ boolean │ varchar │ varchar │ varchar │ varchar │ varchar │ ├─────────┼─────────┼─────────┼─────────┼─────────┼──────────┤ │ true │ compile │ compile │ OK │ │ sudoku_c │ └─────────┴─────────┴─────────┴─────────┴─────────┴──────────┘ Run Time (s): real 0.019 user 0.019758 sys 0.000000 memory D SELECT max(luajit_s(sudoku_c, column0)) from (from read_csv(sudoku17.txt,header0) limit 100); ┌───────────────────────────────────────────────────────────────────────────────────┐ │ max(luajit_s(sudoku_c, column0)) │ │ varchar │ ├───────────────────────────────────────────────────────────────────────────────────┤ │ 978654213645231798213879654796518432521347986384926175437182569159463827862795341 │ └───────────────────────────────────────────────────────────────────────────────────┘ Run Time (s): real 0.075 user 0.031685 sys 0.026326 memory D memory D SELECT max(luajit_s(sudoku_c, column0)) from (from read_csv(sudoku17.txt,header0) limit 1000); ┌───────────────────────────────────────────────────────────────────────────────────┐ │ max(luajit_s(sudoku_c, column0)) │ │ varchar │ ├───────────────────────────────────────────────────────────────────────────────────┤ │ 987563214124987653356412987671398542295174836843256179439825761562731498718649325 │ └───────────────────────────────────────────────────────────────────────────────────┘ Run Time (s): real 0.114 user 0.066773 sys 0.012692 memory D SELECT max(luajit_s(sudoku_c, column0)) from (from read_csv(sudoku17.txt,header0) limit 10000); ┌───────────────────────────────────────────────────────────────────────────────────┐ │ max(luajit_s(sudoku_c, column0)) │ │ varchar │ ├───────────────────────────────────────────────────────────────────────────────────┤ │ 987652431164739582253481796375824619842916357619375824728163945431597268596248173 │ └───────────────────────────────────────────────────────────────────────────────────┘ Run Time (s): real 0.483 user 0.466168 sys 0.003213分别使用两种算法的数独求解程序跳舞链的速度比MRV启发式—dfs快得多。又尝试mold的作者编写的chibicc编译器。先从源码编译二进制可执行文件。C:\Users\ltwsl rootDESKTOP-59T6U68:/mnt/c/Users/lt# docker start gcc gcc rootDESKTOP-59T6U68:/mnt/c/Users/lt# docker exec -it gcc bash root6ae32a5ffcde:/# cd /par root6ae32a5ffcde:/par# cd chibicc-main root6ae32a5ffcde:/par/chibicc-main# make cc -stdc11 -g -fno-common -Wall -Wno-switch -c -o codegen.o codegen.c cc -stdc11 -g -fno-common -Wall -Wno-switch -c -o hashmap.o hashmap.c cc -stdc11 -g -fno-common -Wall -Wno-switch -c -o main.o main.c cc -stdc11 -g -fno-common -Wall -Wno-switch -c -o parse.o parse.c cc -stdc11 -g -fno-common -Wall -Wno-switch -c -o preprocess.o preprocess.c cc -stdc11 -g -fno-common -Wall -Wno-switch -c -o strings.o strings.c cc -stdc11 -g -fno-common -Wall -Wno-switch -c -o tokenize.o tokenize.c cc -stdc11 -g -fno-common -Wall -Wno-switch -c -o type.o type.c cc -stdc11 -g -fno-common -Wall -Wno-switch -c -o unicode.o unicode.c cc -stdc11 -g -fno-common -Wall -Wno-switch -o chibicc codegen.o hashmap.o main.o parse.o preprocess.o strings.o tokenize.o type.o unicode.o root6ae32a5ffcde:/par/chibicc-main# cd ..后面的步骤和tcc一样root6ae32a5ffcde:/par# /par/chibicc-main/chibicc -shared -fPIC -o libsudoku_c4.so dlxdll.c ld: warning: /tmp/chibicc-RZBje2: missing .note.GNU-stack section implies executable stack ld: NOTE: This behaviour is deprecated and will be removed in a future version of the linker root6ae32a5ffcde:/par# ./duckdb155 DuckDB v1.5.5 (Variegata) Enter .help for usage hints. memory D force install luajit from community; memory D load luajit; memory D SELECT * FROM luajit_module(mode : compile, sql_name : sudoku_c, source : return function(p) local ffi require(ffi) ffi.cdef[[ int sudoku_solve(const char *p, char *out); ]] local lib ffi.load(/par/libsudoku_c4.so) local buf ffi.new(char[82]) local rc lib.sudoku_solve(p, buf) if rc ~ 1 then return NULL end return ffi.string(buf) end ); ┌─────────┬─────────┬─────────┬─────────┬─────────┬──────────┐ │ ok │ mode │ phase │ message │ detail │ sql_name │ │ boolean │ varchar │ varchar │ varchar │ varchar │ varchar │ ├─────────┼─────────┼─────────┼─────────┼─────────┼──────────┤ │ true │ compile │ compile │ OK │ │ sudoku_c │ └─────────┴─────────┴─────────┴─────────┴─────────┴──────────┘ memory D SELECT luajit_s(sudoku_c, 000001002000020030004500600007600050080090006100005800001004000070900003400030020); ┌───────────────────────────────────────────────────────────────────────────────────────────────────────────┐ │ luajit_s(sudoku_c, 000001002000020030004500600007600050080090006100005800001004000070900003400030020) │ │ varchar │ ├───────────────────────────────────────────────────────────────────────────────────────────────────────────┤ │ 359461782716829534824573619947682351583197246162345897631254978275918463498736125 │ └───────────────────────────────────────────────────────────────────────────────────────────────────────────┘ memory D .timer on memory D SELECT max(luajit_s(sudoku_c, column0)) from (from read_csv(sudoku17.txt,header0) limit 100); ┌───────────────────────────────────────────────────────────────────────────────────┐ │ max(luajit_s(sudoku_c, column0)) │ │ varchar │ ├───────────────────────────────────────────────────────────────────────────────────┤ │ 978654213645231798213879654796518432521347986384926175437182569159463827862795341 │ └───────────────────────────────────────────────────────────────────────────────────┘ Run Time (s): real 0.065 user 0.026620 sys 0.029178 memory D SELECT max(luajit_s(sudoku_c, column0)) from (from read_csv(sudoku17.txt,header0) limit 1000); ┌───────────────────────────────────────────────────────────────────────────────────┐ │ max(luajit_s(sudoku_c, column0)) │ │ varchar │ ├───────────────────────────────────────────────────────────────────────────────────┤ │ 987563214124987653356412987671398542295174836843256179439825761562731498718649325 │ └───────────────────────────────────────────────────────────────────────────────────┘ Run Time (s): real 0.140 user 0.125244 sys 0.003991 memory D SELECT max(luajit_s(sudoku_c, column0)) from (from read_csv(sudoku17.txt,header0) limit 10000); ┌───────────────────────────────────────────────────────────────────────────────────┐ │ max(luajit_s(sudoku_c, column0)) │ │ varchar │ ├───────────────────────────────────────────────────────────────────────────────────┤ │ 987652431164739582253481796375824619842916357619375824728163945431597268596248173 │ └───────────────────────────────────────────────────────────────────────────────────┘ Run Time (s): real 1.157 user 2.327257 sys 0.011661chibicc编译出来的动态库比tcc编译的动态库执行速度慢一些使用意义不大。上述2种编译器都没有优化再改用gcc -O3优化编译memory D .system bash root6ae32a5ffcde:/par# gcc -shared -fPIC -o libsudoku_c5.so dlxdll.c -O3 root6ae32a5ffcde:/par# exit exit memory D SELECT * FROM luajit_module(mode : compile, sql_name : sudoku_c, source : return function(p) local ffi require(ffi) ffi.cdef[[ int sudoku_solve(const char *p, char *out); ]] local lib ffi.load(/par/libsudoku_c5.so) local buf ffi.new(char[82]) local rc lib.sudoku_solve(p, buf) if rc ~ 1 then return NULL end return ffi.string(buf) end ); ┌─────────┬─────────┬─────────┬─────────┬─────────┬──────────┐ │ ok │ mode │ phase │ message │ detail │ sql_name │ │ boolean │ varchar │ varchar │ varchar │ varchar │ varchar │ ├─────────┼─────────┼─────────┼─────────┼─────────┼──────────┤ │ true │ compile │ compile │ OK │ │ sudoku_c │ └─────────┴─────────┴─────────┴─────────┴─────────┴──────────┘ Run Time (s): real 0.008 user 0.010392 sys 0.000509 memory D SELECT max(luajit_s(sudoku_c, column0)) from (from read_csv(sudoku17.txt,header0) limit 10000); ┌───────────────────────────────────────────────────────────────────────────────────┐ │ max(luajit_s(sudoku_c, column0)) │ │ varchar │ ├───────────────────────────────────────────────────────────────────────────────────┤ │ 987652431164739582253481796375824619842916357619375824728163945431597268596248173 │ └───────────────────────────────────────────────────────────────────────────────────┘ Run Time (s): real 0.458 user 0.438527 sys 0.001587和tcc的结果没有明显差距。瓶颈好像不在动态库这边。