Presto-Kudu-Connector-Support-Array

Presto-Kudu-Connector-Support-Array

Presto是支持Array数据类型的,由于上层应用需要用到Array数据类型,所以当使用不支持Array数据类型的Kudu 作为存储引擎时,需要改造下Kudu 的连接器:

主要的改造思路就是,insert时将数组转换为字符串进行存储,读取时将字符串转换为数组给Presto。

  1. 从数据库中查询出哪些列是array 和 decimal
  2. 在KuduMetadata中修改getColumnHandles、getTableMetadata、beginInsert 方法,使相应的列名支持array-decimal
  3. 修改 KuduPageSink 的appendColumn方法,如果是array 类型,将其转换为特殊字符分割的字符串存入列中
  4. 修改KuduRecordCursor 的getObject方法,如果当前field类型是array,那么将获取的值反向分割转成字符串
  5. 修改NativeKuduClientSession的addConstraintPredicates方法,支持decimal条件过滤

测试:

1
{"kudu.client.master-addresses":"presto-1:7051,presto-2:7051,presto-3:7051","kudu.schema-emulation.enabled":"true","kudu.client.default-socket-read-timeout":"600s","kudu.extension.array.enable":"true"}
1
{"kudu.client.master-addresses":"presto-1:7051,presto-2:7051,presto-3:7051","kudu.schema-emulation.enabled":"true","kudu.client.default-socket-read-timeout":"600s","kudu.extension.array.enable":"true","trino.extension.jdbc.enable":"true","trino.extension.jdbc.driver":"com.mysql.jdbc.Driver","trino.extension.jdbc.url":"jdbc:mysql://10.128.1.94:33061","trino.extension.jdbc.username":"test","trino.extension.jdbc.password":"test","trino.extension.jdbc.max-pool-size":"1"}
1
2
3
4
5
insert into kudu.db_test.event_x(day,user_id,oper_time,oper,xwho,oper_id,buckest_id,"$platform",is_pay,amount,pids) values (20210809,1,16511111112,'click','shenlg',10,1,'ios',true,100.000,ARRAY['pid02','pid03']);

insert into kudu.db_test.event_x(day,user_id,oper_time,amount) values (20210809,1,16511111111,5000.012);

insert into kudu.db_test.event_x(xwho) values ('shujm') WHERE day=20210809 and user_id=1 and oper_time=16511111111;

数组能正常插入,OK


Comments

Your browser is out-of-date!

Update your browser to view this website correctly. Update my browser now

×