Presto-Kudu-Connector-Support-Array
Presto是支持Array数据类型的,由于上层应用需要用到Array数据类型,所以当使用不支持Array数据类型的Kudu 作为存储引擎时,需要改造下Kudu 的连接器:
主要的改造思路就是,insert时将数组转换为字符串进行存储,读取时将字符串转换为数组给Presto。
- 从数据库中查询出哪些列是array 和 decimal
- 在KuduMetadata中修改getColumnHandles、getTableMetadata、beginInsert 方法,使相应的列名支持array-decimal
- 修改 KuduPageSink 的appendColumn方法,如果是array 类型,将其转换为特殊字符分割的字符串存入列中
- 修改KuduRecordCursor 的getObject方法,如果当前field类型是array,那么将获取的值反向分割转成字符串
- 修改NativeKuduClientSession的addConstraintPredicates方法,支持decimal条件过滤
测试:
1 | {"kudu.client.master-addresses":"presto-1:7051,presto-2:7051,presto-3:7051","kudu.schema-emulation.enabled":"true","kudu.client.default-socket-read-timeout":"600s","kudu.extension.array.enable":"true"} |
1 | {"kudu.client.master-addresses":"presto-1:7051,presto-2:7051,presto-3:7051","kudu.schema-emulation.enabled":"true","kudu.client.default-socket-read-timeout":"600s","kudu.extension.array.enable":"true","trino.extension.jdbc.enable":"true","trino.extension.jdbc.driver":"com.mysql.jdbc.Driver","trino.extension.jdbc.url":"jdbc:mysql://10.128.1.94:33061","trino.extension.jdbc.username":"test","trino.extension.jdbc.password":"test","trino.extension.jdbc.max-pool-size":"1"} |
1 | insert into kudu.db_test.event_x(day,user_id,oper_time,oper,xwho,oper_id,buckest_id,"$platform",is_pay,amount,pids) values (20210809,1,16511111112,'click','shenlg',10,1,'ios',true,100.000,ARRAY['pid02','pid03']); |
数组能正常插入,OK