Export Substack Posts to Clean Markdown for RAG

Summary

Pixel shares a guide to exporting Substack posts as clean Markdown for RAG and LLM workflows using an Apify tool and Substack’s public JSON API. It covers cursor-based archive retrieval, post-body requests, and Python integration, with no browser or residential proxies required.

DEV Community

How to export Substack posts to clean Markdown for RAG (no browser needed)

If you have tried to feed a Substack newsletter into an LLM or a RAG pipeline, you have probably hit the same three problems I did: The post HTML is full of subscribe buttons, share widgets and nested image wrappers that pollute your chunks. Substack's archive endpoint no longer accepts offset. It returns HTTP 400, and you have to follow a cursor instead. The archive list does not include the post body, so you need a second request per post. I got tired of rewriting this, so I built a small tool for it and published it on Apify. Disclosure: I am the author. Substack Scraper - Markdown Posts, Search & Comments It uses plain HTTP requests against Substack's public JSON API. No headless browser and no residential proxies, so runs are fast and cheap. 1. Give it a publication Open the Actor, paste this input and press Start: {
"publications": ["astralcodexten"],
"maxPosts": 20,
"sinceDate": "30 days"
}
publications accepts a name, a name.substack.com URL or a custom domain. You can also set searchQuery to find publications about a topic. 2. Get Markdown back Each post is one dataset item. The markdown field starts with the title as an H1, so every document is self-contained when you chunk it: {
"title": "AI Company Employees Should Unionize",
"url": "https://www.astralcodexten.com/p/ai-company-employees-should-unionize",
"publishedAt": "2026-11-12T23:09:15.486Z",
"authors": [{ "name": "Scott Alexander", "handle": "astralcodexten" }],
"wordCount": 1273,
"markdown": "# AI Company Employees Should Unionize\n\nI.\n\nShould safety-conscious employees at AI companies quit in protest, or stay and fight? ..."
}
Headings, lists, code blocks, images with captions and footnotes are kept as standard Markdown. Subscribe and share widgets are removed. 3. Call it from Python from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("gunpark/substack-scraper").call(
run_input={"publications": ["astralcodexten"], "maxPosts": 20}
)

for post in client.dataset(run["defaultDatasetId"]).iterate_items():
print(post["title"], len(post["markdown"]))
From here you can pass post["markdown"] straight to your text splitter and embedding model. Keeping an index up to date Turn on incremental and schedule the Actor. The first run sets a baseline; later runs return only posts published since the previous run, so you never embed the same post twice. Good to know Paywalled posts only expose a public preview. They are skipped by default, or delivered flagged as incomplete if you ask for previews. Comments are optional (includeComments). Pricing is pay per delivered post: $1.20 per 1,000 posts. Failed requests, duplicates and skipped paywalled posts are not charged. The posts belong to their authors. Use the output in ways their terms and copyright allow. If something breaks or you need a field that is missing, open an issue on the Actor page and I will look at it.
Export Substack Posts to Clean Markdown for RAG
Powered by Readybot.io

The latest from Pixel

Incident Response: Look Beyond the Red Alert

Pixel’s DEV Community announcement explores how teams can fixate on one failing service while overlooking the real cause of an outage, such as a full …

Sach: AI Fact-Checks WhatsApp Forwards in India

Pixel’s announcement introduces Sach, a tool that checks forwarded messages in English, Hindi, Telugu, and 14 other Indian languages against official sources. It gives cited …

Java 25 Removes SecurityManager: What to Fix

Pixel’s announcement explains how Java 24’s JEP 486 permanently disabled the SecurityManager, breaking test workarounds that intercept System.exit. It recommends returning exit codes or throwing …