Feature request: integration with Jupyter for LLM support in observable framework code cells

[jaanli]

Hi all! I’ve been preparing course materials for next semester centered on AI at the edge.

For the observable framework modules, there is too much copy and pasting right now.

To minimize this teaching overhead (we can’t afford TAs), I’m exploring Jupyter AI.

here’s a demo:

https://colab.research.google.com/github/onefact/electronic-health-records/blob/main/notebooks/loading_physionet_mimic_data.ipynb

for the other observable framework projects I’ve done, I’ve copy and pasted input and output to claude.ai.

but that’s infeasible for the class, because it adds another dependency such as maccy.app for clipboard history; and leads to high cognitive load (in user experience research for the learning journey).

it would be nice if observable framework supported something similar to ease the onboarding process.

Does that make sense? What would be the engineering lift involved?

I have some spare cycles and can see whether this may be feasible. Next I will be looking at adding WASM based approximate nearest neighbor support for fine tuned LLM applications across sectors: #1130

I will also be giving a talk at duckdb.org conference 2024 later this summer and would love to feature observable framework if possible!!

[Fil]

I've moved this to a discussion. Maybe we can identify practical ways that Framework could do to support this use case?

If you have a program that calls a LLM, then there is a chance that Framework could run it as a data loader.

FWIW I've experimented a bit with dall·e: the following dataloader read a prompt from the same markdown file that calls it, and returned an image:

in dalle.png.js

import "dotenv/config";
import OpenAI from "openai";
import {readFileSync} from "node:fs";

const openai = new OpenAI({apiKey: process.env.OPENAI_API_KEY});
const prompt = readFileSync("./src/dalle.md", "utf-8").match(/prompt: (.*)/)[1];
const image = await openai.images.generate({prompt, n: 1, size: "512x512"});
console.log(JSON.stringify(image.data));

and in dalle.md

prompt: a flower

```js
const image = FileAttachment("dalle.png").json()
```

```js
htl.html`<img src=${m[0].url}>`
```

(I haven't tried to run it recently though.)