how to get List name of listRDD?

[zhxchnl]

In the doc, val listRDD = sc.fromRedisList("keyPattern*") can get Lists in redis, but it get RDD[string] which contains lists only. How to get the corresponding listname of every row of the listRDD?

[fe2s]

Hi @zhxchnl ,
It looks like it is not possible. We will need to create a new API function for this.
For now you can add the following to your project:

class RedisListWithNameRDD(prev: RDD[String], implicit val readWriteConfig: ReadWriteConfig) extends RDD[(String, Seq[String])](prev) with Keys {

  override def getPartitions: Array[Partition] = prev.partitions

  override def compute(split: Partition, ctx: TaskContext): Iterator[(String, Seq[String])] = {
    import scala.collection.JavaConverters._

    val partition: RedisPartition = split.asInstanceOf[RedisPartition]
    val sPos = partition.slots._1
    val ePos = partition.slots._2
    val nodes = partition.redisConfig.getNodesBySlots(sPos, ePos)
    val keys = firstParent[String].iterator(split, ctx)
    groupKeysByNode(nodes, keys).flatMap { case (node, nodeKeys) =>
      val conn = node.endpoint.connect()
      val listKeys = filterKeysByType(conn, nodeKeys, "list")
      val res = listKeys.map(k => (k, conn.lrange(k, 0, -1).asScala)).iterator
      conn.close()
      res
    }
  }
}

Usage:

    val redisConfig = RedisConfig.fromSparkConf(sc.getConf)
    val rwConfig = ReadWriteConfig.fromSparkConf(sc.getConf)
    val keysRdd = new RedisKeysRDD(sc, redisConfig, rwConfig, "keyPattern*")
    val listsWithName = new RedisListWithNameRDD(keysRdd, rwConfig)
    listsWithName.foreach(println)

[gobertm]

Hello @fe2s ,

I would also like to retrieve list names and values while providing a key pattern, but in my case in Java.
I'm able to create the scala class you provided but I'm a little bit lost regarding the usage in a Java context (I am not familiar with scala) and how it can fit with existing java code Spark-redis usage.

Would you be able to provide example of how I can use this code?
PS : I would also be glad if you have code for similar requirement regarding hashes (retrieving the hash name).

Thanks
Maxime