other backends?

[sebastiansturm]

Hi,

this seems to be an awesome package, but I'd really like to use it with the julia kernel instead. Do you plan to support alternative backends, or could you give me a brief overview which parts of your code I'd have to change to add this myself?
thanks,
Sebastian

[ajsteven130]

I think that if you installed iJulia according to the instructions here you just need to modify the custom variable ob-ipython-kernel-extra-args to have the value "--profile julia" (with the quotes). This would add the --profile julia flag to the function below that is used to start the kernel:

(defun ob-ipython--kernel-cmd (name)
  (-concat (list "ipython" "kernel" (format "--IPKernelApp.connection_file=emacs-%s.json" name))
           ob-ipython-kernel-extra-args))

You can modify that variable by either doing M-x customize-group ob-ipython or by setting the variable in your init.el as (setq ob-ipython-kernel-extra-args "--profile julia")

[gregsexton]

I think that should be (setq ob-ipython-kernel-extra-args '("--profile" "julia")).

You'll probably also want to do something like: (add-to-list 'org-src-lang-modes '("ipython" . julia)), if you have a julia-mode. This causes the editing buffer to open in the correct mode.

I don't think it would be a lot of effort to introduce a customisation mechanism that would allow you to choose different backends. I don't necessarily have any plans to add this but I'm not against it either. I've never tried to use ipython with a different backend. Is it as simple as just passing a different parameter?

[sebastiansturm]

thanks alot for your quick response! Unfortunately when I set ob-ipython-kernel-extra-args to '("--profile" "julia")), Emacs never gets past Contacting host: localhost:9988. I checked *ob-ipython-kernel-default*, but that looks inconspicuous to me:

NOTE: When using the `ipython kernel` entry point, Ctrl-C will not work.

To exit, you will have to explicitly quit this process, by either sending
"quit" from a client, or using Ctrl-\ in UNIX-like environments.

To read more about this, see https://github.com/ipython/ipython/issues/2049


To connect another client to this kernel, use:
    --existing emacs-default.json --profile julia

Is there any other way to debug what is going on behind the scenes?

Also, I noticed that when calling ipython kernel --profile julia --debug on the commandline, it seems to start up as expected, but when I try to use that kernel via ipython console --existing ... --profile julia, all input still gets sent to Python. I guess this problem is unrelated, but would still have to be fixed on the Julia side for ob-ipython to work-is that correct?

[ajsteven130]

I'm getting the same thing as @sebastiansturm. ipython kernel behaves differently than most of the other ipython entry points. The ipython console works for me with either julia or R if I start with ipython console --kernel=ir or ipython console --kernel=julia-0.3 but trying to get any of this working with ipython kernel ends in misery.

I can connect other clients to the kernels produced by running ipython console --kernel=*** and they behave properly

[ajsteven130]

I was able to play around with this today. The command issued here ipython kernel is specific to the python kernel, so there can't be any other backends used.

To get things working properly with other backends ob-ipython.el would probably need to write a small kernel manager using a class jupyter_client. Using jupyter_client, I can get kernels started with any backend I have installed and connect to them, but I haven't had time to try integrating with ob-ipython.el.

[sebastiansturm]

great, thanks! I also opened a similar issue on the IJulia github page, in response to which @Carreau pointed me to the Hydrogen source code (https://github.com/willwhitney/hydrogen/blob/master/lib/kernel.coffee#L47-L490). Hydrogen seems to be similar in functionality to ob-ipython (but targets Atom instead of Emacs) and already supports different kernels.

[izahn]

Just want to say that supporting other jupyter kernels would be a killer feature. Jupyter already supports a lot of languages[1], it would be great to have consistent interfaces to them in emacs.

[1] https://github.com/ipython/ipython/wiki/IPython%20kernels%20for%20other%20languages

[ajsteven130]

I created a little kernel starter that has essentially the same functionality as the ipython kernel command used to start the kernel in ob-ipython.el. The key difference is that it takes two optional flags. -f connection_file.json (which exists in ipython kernel) and -k kernel_name (which does not exist in ipython kernel). You can try it out from this gist

For example, to start an R kernel at the command line, you would type python kernel-starter.py -f rkernel.json -k ir. Here ir is the name of the Rkernel if you installed it from here. The main issue with this implementation is that it will only delete the connection_file if it is stopped gracefully, either by KeyboardInterrupt or by sending an http GET request to localhost:port/kill. The port is randomly selected at run time so that multiple kernels can start. The port number being used will show up in the output, so it is easy to find. For the latest stable version of the Julia kernel, you would use -k julia-0.3.

To connect an ipython session to the kernel in the example above, just type ipython console --existing rkernel.json and be careful that the path is correct to rkernel.json

to see possible kernel names that will run, use ipython kernelspec list

I have had trouble integrating it into ob-ipython.el in place of ipython kernel as the kernel starter command, but I have not idea where the problem is. Someone else who understands it better should give it a shot. It may have to do with starting another process, in addition to driver.py, that runs a tornado server. It may be something else local to my system though, as the ob-ipython install from melpa is now hanging when it tries to contact the server started by driver.py.

The long term solution to integrating other backends and multiple kernels is probably to implement a kernel server using ipython/jupyter's MultiKernelManager class, but that would require changing more of ob-ipython because the individual kernels would not run in their own emacs process, and it seems like this would break the kernel detection mechanism in ob-ipython. The benfit would be that you could control everything via http, similar to the implementation here of driver.py

[Carreau]

The python kernel-starter.py -f rkernel.json -k ir seem roughly what we want to to with "kernels nanny" at some point in IPtyhon to handle some edge cases on remote machines.

The second thing does already exist, It's called the notebook server. The notebook server does not only serve notebook, it provides a REST api to start/stop/switch/restart kernels.

[stardiviner]

I'm curious about the process what's going on?

[ilysym]

There is a pull request #38. It would be nice if you try it and report results.

[stardiviner]

I tried this:

#+NAME: matplotlib
#+BEGIN_SRC ipython :session
%matplotlib inline

import matplotlib
import matplotlib.pyplot as plt
import numpy as np
#+END_SRC

#+BEGIN_SRC ipython :session :file /tmp/image.png :exports both :kernel ijulia
plt.hist(np.random.randn(20000), bins=200)
#+END_SRC

It works ok.

[gregsexton]

Merged. Basic support for this is now available. See #57 for more enhancements.