Launch PySpark Jobs such that Python launches the JVM

[ifilonenko]

Current PySpark implementation launches python and the PY4J gateway from the JVM which causes issues from a logging perspective. We should launch the JVM from Python instead. This will require a modification of the Docker file.

[ifilonenko]

@sahilprasad do you think that the solution to this issue would have a problem with your implementation of launching jupyter-notebooks from client mode

[sahilprasad]

@ifilonenko I'm not sure if it would cause issues. Currently, I can just run Jupyter on its own and run PySpark applications on top of it that communicate with k8s. Because of this, I'm not sure if a PySpark app launching the JVM would necessarily conflict with this method of execution, but this does seem like something we should be able to work around in the case that it does not work.