For AI agents: docs.octen.ai/capabilities/extract.md
LLMs and agents are constantly pointed at specific pages: a link pasted into the chat, a source cited in a document, a set of URLs to ingest into a knowledge base. But a web page is built for browsers, not models; the content that matters is buried in markup, navigation, ads, and scripts. Octen Extract turns URLs into clean, LLM-ready content: the main text of each page, parsed into markdown or plain text, labeled with what the page is (page_structure) and what domain it belongs to (category).
For the full list of parameters, see the Extract API reference.
Why Extract
- Read what the task points to. A pasted link, a cited source, a referenced doc: when the task names the page, the agent has to read that exact URL.
- Ingest knowledge. Turn a known set of URLs, such as docs, wikis, and blogs, into clean text for RAG indexing.
- Keep answers current. Re-read the specific pages that matter, such as pricing pages, docs, and changelogs, whenever they change.
Why Octen
- LLM-ready clean markdown. Every page parses into structured markdown, ready to drop into RAG pipelines and agents. PDFs are read natively.
- Page understanding built in. Every result is labeled with its page type and a content category drawn from a taxonomy of 160+ categories, so login walls, error pages, and other non-content can be filtered before they reach the model.
- Multimodal asset extraction. One request returns the text plus the page’s images, videos, and audio, made for multimodal agents and RAG.
- Success-only billing. Up to 20 URLs per request, each succeeding or failing independently, and only successful URLs are billed, at a flat $1 per 1,000 successful URLs.
How It Works
- Send up to 20 URLs.
- Octen fetches each page, classifies what it is, and strips it to the main content.
- The response returns one result per URL, with its content, its page labels, and a success or failure status.
Scenarios
Read pages
Fetch one or more URLs as clean markdown.Skip pages that aren’t content
Every result carries the labels by default; no extra parameter is needed. Checkpage_structure before the content reaches the model: pages labeled No Main Content, such as login walls and error pages, can be dropped without spending tokens on them, and category tells the agent what domain each remaining page belongs to.
Extract only the relevant parts
Long pages can flood a model’s context with content the task never needed. Pass intent-focused keywords to return query-relevant highlights for each URL instead of the complete content.Force fresh content
Lower the maximum cache age when the page changes frequently.Collect media
Return the image, video, and audio URLs found on each page.Next Steps
Try in Console
Run Extract live in the Octen console.
Extract API Reference
Full request/response schema.