Kudu表复制(表数据迁移)

Kudu表复制(表数据迁移)

Kudu 表复制(表数据迁移)

https://kudu.apache.org/docs/command_line_tools_reference.html#table-copy

注意事项:

  1. 要求复制表于原表的列的结构一致(不要求分区和副本一致,所以可以调整表的副本数)
  2. 可以先只复制表结构
  3. predicates 预言 大小比值 只支持数值类型,时间戳类型不支持
  4. 注意超时设置
  5. write_type 参数有3种:
    1. 空字符 ,表示只复制表结构
    2. insert,直接插入
    3. upsert,更新插入
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
# 复制表结构,不含数据,完全一样
kudu table copy 10.128.2.162:7051,10.128.2.72:7051,10.128.2.172:7051 bn_op 10.128.2.162:7051,10.128.2.72:7051,10.128.2.172:7051 -create_table=true -dst_table=bn_op_1228 -num_threads=8 -write_type=

#I1228 15:49:41.570058 3034728 table_scanner.cc:466] Table bn_op_1228_2 created successfully


# 复制数据(要求表结构一致)
kudu table copy 10.128.2.162:7051,10.128.2.72:7051,10.128.2.172:7051 bn_op 10.128.2.162:7051,10.128.2.72:7051,10.128.2.172:7051 -nocreate_table -dst_table=bn_op_1228 -num_threads=8 -write_type=insert -timeout_ms=600000

# predicates,不支持时间戳... 所以用时间做分区时,最好同时存一份整型的日期
kudu table copy 10.128.2.162:7051,10.128.2.72:7051,10.128.2.172:7051 bn_op 10.128.2.162:7051,10.128.2.72:7051,10.128.2.172:7051 -nocreate_table -dst_table=bn_op_1228 -num_threads=8 -write_type=insert -timeout_ms=600000 -predicates='["AND", [">=", "timestamp", 1669824000000000]]'

# 整型过滤后再复制表数据
kudu table copy 10.128.2.162:7051,10.128.2.72:7051,10.128.2.172:7051 bn_op 10.128.2.162:7051,10.128.2.72:7051,10.128.2.172:7051 -nocreate_table -dst_table=bn_op_1228 -num_threads=8 -write_type=insert -timeout_ms=600000 -predicates='["AND", [">=", "gbl_log_2", 800], ["NOTNULL", "gbl_log_3"]]'


#kudu table copy 10.128.2.162:7051,10.128.2.72:7051,10.128.2.172:7051 bn_op 10.128.2.162:7051,10.128.2.72:7051,10.128.2.172:7051 -nocreate_table -dst_table=bn_op_1228 -num_threads=8 -write_type=insert -timeout_ms=600000
#I1228 16:03:46.345592 3038583 table_scanner.cc:668] Scanned count: 100792
#I1228 16:03:51.345794 3038583 table_scanner.cc:668] Scanned count: 216486
#。。。。
#Total count 1435409 cost 111.099 seconds

 

Comments

Your browser is out-of-date!

Update your browser to view this website correctly. Update my browser now

×