【背景】业务需求是通过doris异步物化视图定时查询paimon的数据进行结果更新加速报表查询速度。【版本】doris 3.1.4paimon 1.4.1【问题】doris子线程持续增长导致内存同步增长最终宕机【复现步骤】1、创建paimon catalogCREATE CATALOG if not EXISTS paimon_catalog PROPERTIES (warehouse s3a://xxxxxx,type paimon,s3.secret_key xxxxxxx,s3.region ap-east-1,s3.endpoint s3.ap-east-1.amazonaws.com,s3.access_key xxxxx);2、创建异步物化视图定时调度CREATE MATERIALIZED VIEW app_order_hourBUILD IMMEDIATE REFRESH AUTO ON SCHEDULE EVERY 30 MINUTEPROPERTIES (replication_num 1)AS SELECT xxx from paimon_catalog.etl.tb3、通过排查发现JNI的子线程数量一直上涨【解决办法】通过和doris社区的专家沟通让我加上S3的一些配置最终测试有效的配置如下重建catalogCREATE CATALOG if not EXISTS paimon_catalog PROPERTIES ( warehouse s3a://xxxxx, type paimon, s3.secret_key xxxxx, s3.region ap-east-1, s3.endpoint s3.ap-east-1.amazonaws.com, s3.access_key xxxxx, fs.s3a.access.key xxxxx, fs.s3a.secret.key xxxxx, fs.s3a.endpoint.region ap-east-1, fs.s3a.endpoint s3.ap-east-1.amazonaws.com, fs.s3a.impl org.apache.hadoop.fs.s3a.S3AFileSystem, fs.s3a.impl.disable.cachefalse, fs.s3a.connection.maximum1000 );重启be后持续观察
【背景】业务需求是通过doris异步物化视图定时查询paimon的数据进行结果更新加速报表查询速度。【版本】doris 3.1.4paimon 1.4.1【问题】doris子线程持续增长导致内存同步增长最终宕机【复现步骤】1、创建paimon catalogCREATE CATALOG if not EXISTS paimon_catalog PROPERTIES (warehouse s3a://xxxxxx,type paimon,s3.secret_key xxxxxxx,s3.region ap-east-1,s3.endpoint s3.ap-east-1.amazonaws.com,s3.access_key xxxxx);2、创建异步物化视图定时调度CREATE MATERIALIZED VIEW app_order_hourBUILD IMMEDIATE REFRESH AUTO ON SCHEDULE EVERY 30 MINUTEPROPERTIES (replication_num 1)AS SELECT xxx from paimon_catalog.etl.tb3、通过排查发现JNI的子线程数量一直上涨【解决办法】通过和doris社区的专家沟通让我加上S3的一些配置最终测试有效的配置如下重建catalogCREATE CATALOG if not EXISTS paimon_catalog PROPERTIES ( warehouse s3a://xxxxx, type paimon, s3.secret_key xxxxx, s3.region ap-east-1, s3.endpoint s3.ap-east-1.amazonaws.com, s3.access_key xxxxx, fs.s3a.access.key xxxxx, fs.s3a.secret.key xxxxx, fs.s3a.endpoint.region ap-east-1, fs.s3a.endpoint s3.ap-east-1.amazonaws.com, fs.s3a.impl org.apache.hadoop.fs.s3a.S3AFileSystem, fs.s3a.impl.disable.cachefalse, fs.s3a.connection.maximum1000 );重启be后持续观察
平方复杂度无法被常规优化消除｜TaoToken 统一 Key 通道实测)
