For AI agents: docs.octen.ai/capabilities/extract.md
LLMs and agents are constantly pointed at specific pages: a link pasted into the chat, a source cited in a document, a set of URLs to ingest into a knowledge base. But a web page is built for browsers, not models; the content that matters is buried in markup, navigation, ads, and scripts. Octen Extract turns URLs into clean, LLM-ready content: the main text of each page, parsed into markdown or plain text, labeled with what the page is (`page_structure`) and what domain it belongs to (`category`). For the full list of parameters, see the [Extract API reference](/api-reference/extract). ## Why Extract * **Read what the task points to.** A pasted link, a cited source, a referenced doc: when the task names the page, the agent has to read that exact URL. * **Ingest knowledge.** Turn a known set of URLs, such as docs, wikis, and blogs, into clean text for RAG indexing. * **Keep answers current.** Re-read the specific pages that matter, such as pricing pages, docs, and changelogs, whenever they change. ## Why Octen * **LLM-ready clean markdown.** Every page parses into structured markdown, ready to drop into RAG pipelines and agents. PDFs are read natively. * **Page understanding built in.** Every result is labeled with its page type and a content category drawn from a taxonomy of 160+ categories, so login walls, error pages, and other non-content can be filtered before they reach the model. * **Multimodal asset extraction.** One request returns the text plus the page's images, videos, and audio, made for multimodal agents and RAG. * **Success-only billing.** Up to 20 URLs per request, each succeeding or failing independently, and only successful URLs are billed, at a flat \$1 per 1,000 successful URLs. ## How It Works 1. Send up to 20 URLs. 2. Octen fetches each page, classifies what it is, and strips it to the main content. 3. The response returns one result per URL, with its content, its page labels, and a success or failure status. ## Scenarios ### Read pages Fetch one or more URLs as clean markdown. ```bash theme={null} curl -X POST https://api.octen.ai/extract \ -H "Content-Type: application/json" \ -H "x-api-key: YOUR_API_KEY" \ -d '{ "urls": ["https://example.com/article", "https://example.com/report"], "format": "markdown" }' ``` ### Skip pages that aren't content Every result carries the labels by default; no extra parameter is needed. Check `page_structure` before the content reaches the model: pages labeled `No Main Content`, such as login walls and error pages, can be dropped without spending tokens on them, and `category` tells the agent what domain each remaining page belongs to. ```json theme={null} { "urls": ["https://www.instagram.com/instagram/"] } ``` Each result comes back labeled; unrelated fields are omitted here: ```json theme={null} { "url": "https://www.instagram.com/instagram/", "status": "success", "page_structure": { "primary": "No Main Content", "secondary": "Operation Page" }, "category": { "primary": "Computers, Electronics & Technology", "secondary": "Social Media Networks" }, "full_content": "Log into Instagram ..." } ``` ### Extract only the relevant parts Long pages can flood a model's context with content the task never needed. Pass intent-focused keywords to return query-relevant highlights for each URL instead of the complete content. ```json theme={null} { "urls": ["https://example.com/annual-report"], "query": "revenue growth and guidance" } ``` ### Force fresh content Lower the maximum cache age when the page changes frequently. ```json theme={null} { "urls": ["https://example.com/live-blog"], "max_age_seconds": 300 } ``` ### Collect media Return the image, video, and audio URLs found on each page. ```json theme={null} { "urls": ["https://example.com/gallery"], "include_images": true, "include_videos": true, "include_audio": true } ``` ## Next StepsLast modified on August 22, 2026