Very simple configurable text refinery

[twdragon]

llama.cpp Configurable Text Refinery

Currently a pet project, very simple, configurable fragment-overlapped text refinery, initially developed to process meeting transcripts with Mistral models, in English, Italian and Russian.

https://github.com/twdragon/llama.cpp-text-refinery

Features

• Server autostart (external llama-server instance is also supported via --external-server)
• Config override from CLI
• YAML configuration
• Generation is reconfigurable on a per-model-file basis.
• May use ffmpeg and whisper.cpp to process raw audio/video recordings.
• Time-based screencaps generation support
• Free prompting (the prompt does not contain system parts by default)
• Configurable rolling window and overlapping coefficients from both CLI and configuration file

Full options list

positional arguments:
  input_text            File containing input text/multimedia for refining

options:
  -h, --help            show this help message and exit
  -es, --external-server
                        Use external llama.cpp server instead of running our own
  -o, --output-dir [OUTPUT_DIR]
                        Output directory to store results [input file directory]
  -c, --config [CONFIG]
                        YAML configuration file, the default is C:\Users\andrey.vukolov\src\summarizer\config.yml
  -m, --model [MODEL]   Quantized LLM file to load, overrides config file setting
  -cl, --context-length [CONTEXT_LENGTH]
                        LLM context window length limit [4096]
  -b, --batch-length [BATCH_LENGTH]
                        LLM loader batch length [2048]
  -w, --window-coeff [WINDOW_COEFF]
                        Context window width coefficient to calculate the tokenizer context window, window-width = window-coeff * context-length [0.6714], overrides config file setting
  -ov, --overlap-coeff [OVERLAP_COEFF]
                        Context window overlap coefficient [0.05], overrides config file setting
  -pl, --predict-limit-coeff [PREDICT_LIMIT_COEFF]
                        Prediction limit calculation coefficient [0.4272], overrides config file setting
  --model-remap         Set on model RAM remapping mode (for multiple-instance or multi-GPU servers)
  --enable-webui        Turn on web UI renderer on llama.cpp server
  -gl, --gpu-layers [GPU_LAYERS]
                        Instructs the server to upload a number of LLM layers to GPU VRAM [40]
  -p, --prompt-preset [PROMPT_PRESET]
                        Select the prompt preset instead of the default (first appeared) one
  -lp, --list-presets   List prompt presets, then exit
  -j, --join-text       Join all the generated text into one Markdown document
  -wh, --whisper        Use whisper.cpp utility to recognize speech. Requires whisper.cpp and ffmpeg being installed
  -wl, --whisper-language [WHISPER_LANGUAGE]
                        Select the language preset for whisper.cpp
  --whisper-translate   Set translation mode for whisper.cpp
  -wt, --whisper-keep-txt
                        Do not remove TXT artifact from whisper.cpp
  -rm, --render-markdown
                        Render Markdown to HTML (requires Python Markdown extension package installed)
  --video-frames        Tries to extract preview frames from the input video when rendering Markdown to HTML
  --video-frames-cnt [VIDEO_FRAMES_CNT]
                        Number of the preview frames to generate [2..99]
  --video-frames-width [VIDEO_FRAMES_WIDTH]
                        Preview frame width