Skip to content

taos-jdbcdriver高版本驱动问题导致无法通过spark读取tdengine数据 #272

@jasonTangxd

Description

@jasonTangxd

tdengine版本:3.3.6.9
spark版本:3.3.2
使用taos-jdbcdriver版本:3.6.2
通过spark读取数据即df.show(5), 没有数据,时间戳类型的数据是返回null, string类型的直接返回的字段名,double类型的会报转换错误
不管是用RestfulDriver或者WebSocketDriver驱动都一样
代码如下:
`
SparkSession spark = SparkSession.builder()
.appName("appSparkTest")
.master("local[*]")
.getOrCreate();

// connect
String url      = "jdbc:TAOS-RS://192.168.1.1:6041/test?user=root&password=taosdata";
String driver   = "com.taosdata.jdbc.rs.RestfulDriver";
// String url = "jdbc:TAOS-WS://192.168.1.1:6041/test?user=root&password=taosdata";
// String driver = "com.taosdata.jdbc.ws.WebSocketDriver";
int timeout = 60; // seconds
try {
// query sql
DataFrameReader reader = spark.read()
.format("jdbc")
.option("url", url)
.option("driver", driver)
.option("queryTimeout", timeout);

  // map table
  String dbtable  = "(select ts, mete_code from xxx) as t;";
  Dataset<Row> df = reader.option("dbtable", dbtable).load();
  String log      = String.format("------------ show dbtable read:%s -----------", dbtable);
  System.out.println(log);
  
  // show schema
  df.printSchema();

  // show data
  df.show(5);

`

结果如下:
`
root
|-- ts: timestamp (nullable = true)
|-- mete_code: string (nullable = true)

  +----+---------+
  | ts|mete_code|
  +----+---------+
  |null|mete_code|
  |null|mete_code|
  |null|mete_code|
  |null|mete_code|
  |null|mete_code|
  +----+---------+
  only showing top 5 rows

`

但是把taos-jdbcdriver版本换成3.0.0 使用RestfulDriver驱动是能正常获取数据的
`
root
|-- ts: timestamp (nullable = true)
|-- mete_code: string (nullable = true)

  +-------------------+--------------------+
  |                 ts|           mete_code|
  +-------------------+--------------------+
  |2024-06-27 10:15:00|ACTIVE_THRUST_BRG...|
  |2024-06-27 10:16:00|ACTIVE_THRUST_BRG...|
  |2024-06-27 10:18:00|ACTIVE_THRUST_BRG...|
  |2024-06-27 10:20:00|ACTIVE_THRUST_BRG...|
  |2024-06-27 10:22:00|ACTIVE_THRUST_BRG...|
  +-------------------+--------------------+
  only showing top 5 rows

`

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions