@simonw - Three that I use: 1. curl URL | strip-tags
Matt Pocock@mattpocockuk
Best CLI for crawling a URL and turning it into LLM-ready text... hit me
Simon Willison✓@simonw
2025-03-24Three that I use:
1. curl URL | strip-tags https://github.com/simonw/strip-tags
2. curl 'http's://r.jina.ai/http..url-here https://jina.ai/reader/ (Reader API - Jina AI — Convert any URL to Markdown for better grounding LLMs.)
3. shot-scraper with Readability.js https://shot-scraper.datasette.io/en/stable/javascript.html (Scraping pages using JavaScript - shot-scraper)
shot-scraper javascript https://simonwillison.net/2022/Mar/24/datasette-061/ ” async () => { const readability = await import(‘https://cdn.skypack.dev/@mozilla/readability'); return (new readability.Readability(document)).parse(); }”
The output looks like this:
{ “title”: “Datasette 0.61: The annotated release notes”, “byline”: null, “dir”: null,” loading=“lazy”>