IDE fails to see spark.implicits._ import

[LvffY]

Hi everyone

First of all, thanks for this repo ❤️ !

Context

We're trying to use Databricks connect as a backend for our Unit tests process. It means that our SparkSession during our test will be a DatabricksSession and connect to a real Databricks cluster to improve our tests accuracy.

We're using the Spark Scala API.

In this API, we want to use codes like :

import spark.implicits._
df.withColumn("myNewCol", $"myOldCol" * 2)

Unfortunately, we encouter some issues when using latest versions of Databricks Connect with Intellij.

Problem

I've made a dedicate fork to illustrate my point. If needed, you can see all changes here.

In a nutshell :

We've update the FilterInvalid object to look like this

object FilterInvalid {
  def run(df: DataFrame, spark: SparkSession): DataFrame = {
    import spark.implicits._
    df.filter($"distance" > 0)
      .filter($"amount" > 0)
      .filter(!(($"distance" < 5) && ($"amount" > 100)))
  }
}

However, when using any databricks-connect dependency >= 17.0.3, in Intellij se have issues with this imports that is not recnognized :

If we're using any databricks-connect dependency <= 16.4.7, the imports is well recognized :

Notes

This is only a "compile" issue. If we run the code, this is working fine independently of the databricks-connect version we're using.