Using 'reeder' to textify more kinds of web resources for llm workflows in the terminal

[rpdelaney]

Simon already provided strip-tags for reducing HTML to plain text. If you just need to extract plain text for use in your cli workflows, you should stick with that.

However, my use cases are a bit more demanding, as I also use LLMs to interrogate PDF documents and YouTube videos. So, I'm developing reeder to fill those gaps. My aim is to have a one-stop-shop command-line tool for textifying arbitrary web content.

Here's a simple example of reeder summarizing a news story:

$ reed 'https://www.wired.com/story/2025-4chan-hack-admin-leak/' | llm prompt -t tldr -m 4o-mini 

Markdown output

4chan Faces Potential Data Breach After Speculated Hack

4chan, an anonymous image board known for enabling controversial content, is under scrutiny following rumors of a hack that may have exposed sensitive data of its administrators and moderators. Initial reports of a breach emerged after a previously banned board briefly reappeared online, accompanied by indications that hackers had defaced the site. If confirmed, this hack could significantly undermine the anonymity of the platform and its users, who may have mistakenly believed their data was secure. Experts suggest that if the breach is genuine, it could have considerable repercussions, given the site's long-standing controversial history and potential implications for law enforcement.

Key facts

• Speculation about a hack on 4chan arose after a series of outages and reports of a site defacement with the message, "U GOT HACKED XD."
• Allegations surfaced that hackers accessed 4chan’s backend systems, revealing usernames and email addresses of moderators and administrators.
• Concerns have been raised that the breach could expose personal information about users who registered on the platform, potentially including high-profile individuals.
• Experts note that 4chan may have vulnerabilities due to outdated software, making it susceptible to attacks.
• 4chan's controversial history includes accusations of harboring hate speech and being linked to violent incidents, leading to increased scrutiny from authorities.
• If the hack's validity is confirmed, it might have implications for user anonymity and law enforcement investigations.
• The longevity and culture of 4chan have made it significant in the realm of online discourse, with potential shifts in platform dynamics if it faces decline.

Or, using reeder and llm to extract contact information into a qrcoded vCard for easy importing to a mobile device:

$ reed "https://friscolibrary.com/contactus/" | llm prompt "Extract only the contact information and format it as a vCard." -x -m 4o-mini | qrencode -t UTF8i -o -

Console output

    █▀▀▀▀▀█ ▄▄█▀ ▀▄▀   ▀▀ ▄█▄▄▀▀ ▀▀▄▄▄  ▄▀▄▀▄▀█▄  █▀▀▀▀▀█
    █ ███ █ ▄█▀▀ ▄▀▀ ▄█▄  ▀▀▄ ▀██▄█▄ ▄█▄▀▄█▀▄▄▄▀▄ █ ███ █
    █ ▀▀▀ █ ▄█▄▀██▄█▀▀▀ ▄▄ ▀█▀▀▀█▄██ ▄ ▀█  █▀▄█   █ ▀▀▀ █
    ▀▀▀▀▀▀▀ ▀ ▀▄█ █ █▄█▄█ ▀▄█ ▀ █ ▀ █ ▀▄█ █▄█▄▀ ▀ ▀▀▀▀▀▀▀
    ▀█▀▀▀▄▀█▀▄ ██▀▀▀▄██▀ ▀▀██▀▀█▀██▀▄▄▀█▀█ ▄▄ ▄▄ ▀▄▀ █▄█▄
     ▄▄▀█▀▀▀▄▀█▄▄▀▀▀█▄ ▀▄█▄▀ █  ▀▄▄ ▄ ▀█  █▀ ▀▄▄▀▄ ██   ▀
      █ ▀▀▀█▀▀████▀▄▄▄█  ▄ █▀▄ █▀▄▀█  ▄ █  █▀▄▀█▄▄▀██ ▄▄▀
    ▄▄█ ██▀▄ █ ▄▀▄ ▄ ▄▄█   ███▄▀ ▀▄ ▄▄▄ ▄ ▀▄▄▀▀▀▀█▄ ▀██
    ██▄ ▀ ▀ ▄▄▄ ▀▄ ▄  █▀▀▄██▀▄ █▀▀▄█ ▀▀▀▀▀▀  ▄▄▄  ▀▀███▄▀
    ▄▀▄▀▄█▀█ ██▄  ▀▄ ▄ █ ▀█ ▀▄ ▀▀▄▄█ █  ▄▄▀   █ █▀▄▄█▄▄▀
     █▀▄ █▀ ▄  ▀▀█▀ ▄▄█ ▄▀▄█▀▀▄ ▄███ ▀▄▄▄▀▄ ███▀▄▀█▄█ ▄▄
    ██▀██ ▀█ ▀▄▄▀▀█▀██▄▀▄▀ ▀▄█▄▄▄ ███▀  █▀ ▀▄▄██ ▄ ▀▀█▄ ▀
    ▄▀  █▀▀▀██ ▄ ▀▀▀▄ █▀ █▀▀█▀▀▀█▄█▀ ▀▀▄█▄ ▄ ▄▄ █▀▀▀██▄▄▀
    █▀ ▄█ ▀ ███▄▄▄█▀█ ▄▀▄██▀█ ▀ █ █ ▄ █▄  ▄▀ █  █ ▀ █▀  ▀
    █▀▄▀██▀▀██ ▀ ▀▀▀ ██▀ ▀▄ ▀▀██▀██▀▀█  ██▀ ▀▄  █████▄▄▄
     ▀▄▄█▀▀▀█▄▄▄▀▄ ▀███    ▀  ▄██▄▀▄ █▀█ ▄█  ▀█ ██▀█▄▀ ▄█
    ▄█▀  ▄▀▄█  ▄ ▄ ▄ ▀█ ▀▄▀ ▀▄▄▄ ▄█▀ ▀▀▀▀▄ ▄ ▄▄▄█  ▀▀██ ▀
    ▀ █ ▄▀▀█  █ █▀█▄ ▀▄ ▄▀█ ██  ▀  ▄▄██ █▄▀▀ ▄▄▀▄▀  ▄▄█ █
    ▄▀█  ▀▀ ▀▄▀ ▄█▀ ▄▀▄  ▀▄▄█  ▄ ▀█▄ ▄  ▀▄▀ ▄▀█ ▄▄▄▀██ ▄▀
    ▀▀ ▄▄▄▀ ▀█▄█ ▀█ ▄▄▄ ▄▄▀▀▄▄█▄█▀▀▀▄▄▄▄▀ █▄█▀▀ ▄▄█ ▀▀ █▄
    ▀ █▀▀█▀ ▄▄ ██▀▀▀▄▄██▄▀█▀ █ ▄▄██▄▄  ███ ▀▄██ ▀▄▄▄█▄
    ▀█▄▀▀▀▀█ █▄▀▀██▀█ ▄█▄▀█ ▄▄ ▀▀▄▄▀█ ▀ █ █▄▀█▄     ██ ▀▀
       ▀  ▀ █ ▄█ ██ ▄▄█▄▀█▄▀█▀▀▀█▀█▄ ▄ ▄ ▄█  ▄▄ █▀▀▀█▀
    █▀▀▀▀▀█ ▀▀█ ██ ██ █▀ ▀▀▄█ ▀ █▀█▀▄▄▀ ▀  █▄▀█▄█ ▀ █▄ ▀
    █ ███ █ █▀▀██▀▄█  ▀▀█▀▀██▀███▄▄█▄ ▀▀▀▄▀ ▄█▄▄▀█▀▀█▄ ▄
    █ ▀▀▀ █ █▀▀██  █▄▀██▄█ ▄▄█ █▄▀   ▀▀▄▀██▀ ▄▄  ▀█  ▄▄█▄
    ▀▀▀▀▀▀▀ ▀▀▀▀▀     ▀▀▀▀ ▀▀   ▀ ▀     ▀▀▀▀  ▀▀ ▀  ▀▀ ▀

reeder can extract subtitles from YouTube:

$ reed 'https://www.youtube.com/watch?v=d-7o9xYp7eE' | llm prompt -t tldr -m 4o

Markdown output

Overview of Why You Should Never Talk to the Police

In a talk aimed to replicate a typical law school classroom experience, emphasis is placed on the importance of the Fifth Amendment, specifically the right not to self-incriminate. The presenter advocates fervently for individuals to never engage in conversation with the police, arguing that it can rarely be advantageous and often leads to negative consequences. Drawing on experiences as a criminal defense attorney and various cases, the discussion provides multiple instances wherein even innocent individuals have inadvertently implicated themselves in criminal investigations by talking to law enforcement. The talk also includes insights from a police detective acknowledging the accuracy of these claims while providing perspective from law enforcement on interrogation techniques.

Key Facts

• The Fifth Amendment provides that no person shall be compelled in any criminal case to be a witness against themselves.
• Engaging with police during an investigation can unintentionally lead to self-incrimination due to the complexities and breadth of criminal statutes.
• Many individuals, regardless of their innocence or guilt, often make statements that police use as confessions or admissions of guilt.
• The speaker argues that anything said to police, even during informal questioning, is potentially admissible in court.
• Innocent individuals have been wrongly convicted based on misinterpreted statements to police and faulty evidence.
• Real-world examples are provided to illustrate the unintentional consequences of talking to police, even when individuals believe they are helping themselves.
• The presenter highlights that police techniques are often designed to elicit confessions or potentially implicating statements, whether intentionally misleading or not.
• The accompanying law enforcement officer corroborates the likelihood of self-incrimination when suspects engage in discussions with police.
• The talk concludes with advice to remain silent and invoke the right to legal counsel when approached by law enforcement in investigative contexts.

Finally, I'm also using reeder to generate briefs on legal documents:

$ reed 'https://storage.courtlistener.com/recap/gov.uscourts.dcd.278436/gov.uscourts.dcd.278436.81.0_5.pdf' | llm prompt -t brief -m o1-mini

JSON output

[
    {
        "citation": "J.G.G. v. Trump, Civil Action No. 1:25-cv-00766-JEB (D.D.C. Apr. 16, 2025)",
        "parties": [
            { "name": "J.G.G.", "role": ["Plaintiff"] },
            { "name": "Donald J. Trump", "role": ["Defendant"] }
        ],
        "narrative": [
            { "type": "fact", "data": "On March 15, 2025, the Court issued a Temporary Restraining Order (TRO) preventing the Government from transferring certain individuals into foreign custody under the Alien Enemies Act." },
            { "type": "fact", "data": "Despite the TRO, the Government proceeded with removal operations, transferring two planeloads of protected individuals to a Salvadoran mega-prison hours after the Order was issued." },
            { "type": "fact", "data": "The plaintiffs filed a lawsuit seeking to restrain the government's removal actions and to certify a class of similarly situated individuals." },
            { "type": "fact", "data": "The Supreme Court later vacated the TRO on procedural grounds, determining that challenges to removal under the Act must be brought through a habeas-corpus proceeding." },
            { "type": "prior_proceeding", "data": "Plaintiffs filed the initial lawsuit and obtained the TRO on March 15, 2025." },
            { "type": "prior_proceeding", "data": "The Government appealed the TRO issuance." },
            { "type": "prior_proceeding", "data": "The Supreme Court vacated the TRO on April 7, 2025." }
        ],
        "facts": [
            { "data": "The TRO was issued to ensure that individuals subject to removal under the Alien Enemies Act could contest their removability in federal court, safeguarding their due process rights." },
            { "data": "The Government violated the TRO by transferring two planeloads of protected individuals to a foreign detention facility shortly after the Order was issued." },
            { "data": "The plaintiffs allege that the Government's actions demonstrate a willful disregard for the Court's Order." }
        ],
        "issue": "Did the Government willfully disregard the Temporary Restraining Order by transferring protected individuals into foreign custody, thereby constituting criminal contempt?",
        "rule": "Under 18 U.S.C. § 401, criminal contempt requires that a court order be clear and specific, that it was violated, and that the violation was willful or deliberate.",
        "application": "The Court found that the TRO issued was clear and specifically prohibited the transfer of protected individuals into foreign custody. The Government proceeded with the removal actions despite the TRO, without providing a satisfactory explanation or attempting to comply. This behavior indicates a willful disregard for the Court's Order, fulfilling the criteria for criminal contempt.",
        "conclusion": "The Court concluded that there is probable cause to find the Government in criminal contempt for willfully violating the Temporary Restraining Order."
    }
]

reeder is under active development: it still has rough edges, lacks docs, and the API should be considered unstable. But if you'd like to give it a try, you can install it from pypi via, e.g. pipx install reeder.

[agustif]

That's cool, you could make an llm-reeder llm plugin for easy use alongside llm with a keyword like

llm reed <url>