> ## Documentation Index > Fetch the complete documentation index at: https://agentheya.mintlify.site/llms.txt > Use this file to discover all available pages before exploring further. # Internal Data > Upload private files that your agent uses for retrieval-augmented generation. **Field reference:** [`data_internal`](/reference/data_internal) — every field, type, default, and tier for this page. Internal Data is private knowledge your agent retrieves to inform its answers. Files you add here are indexed and made searchable. When a user asks a question, the agent retrieves relevant chunks and uses them as context — this is retrieval-augmented generation (RAG). Internal data is never shared with subscribers as downloadable files. It is used only to inform the agent's responses. ## Adding files 1. Open your agent and go to **Internal Data** in the sidebar. 2. Drag and drop files onto the drop zone, or click it to browse. 3. The file uploads and starts indexing automatically — changes on this page save on their own (there is no Save button). Supported file types: text formats (Markdown, JSON, CSV, plain text) plus documents that can be converted to text — **PDF, DOCX, XLSX, HTML** and similar are automatically converted to text and structure before indexing. Executable files and other binaries are rejected. The maximum file size is set by your subscription plan and enforced on upload (both client-side and server-side). ## Indexing status After saving, each file shows a status badge as the background ingest pipeline processes it: | Badge | Meaning | | ----------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Uploading…** | File picked locally, upload in progress. | | **Queued** / **Queued (next)** / **Queued (N ahead)** | Waiting in the ingest queue. | | **Converting…** | The source file is being converted to a searchable text format. | | **Review pending** | Chunking paused for your review (see [Review chunks before ingest](#review-chunks-before-ingest)) — the file waits until you ingest or discard the preview. | | **Processing…** | Chunks are being embedded and written to the retrieval store. | | **Loaded** | Indexed and searchable by the agent. | | **Error: …** | Ingest failed; hover the badge for details. | Once a file is **Loaded**, you can click the info icon to view file stats (chunk count and total tokens). ## Chunking configuration Before a file is indexed it is split into **chunks** — the passages retrieval actually searches and returns. Each file carries its own chunking config, opened from the **Chunking & metadata** control on the file's row. Defaults are sensible for most documents; tune only when retrieval quality is off. Changes take effect the next time the file is (re-)processed. ### Presets Start from a preset, then fine-tune if needed. A preset only sets the three splitting knobs (size, overlap, strategy) — metadata and enrichment settings survive a preset switch. | Preset | Size · overlap · strategy | Use for | | ---------------------------- | -------------------------- | ---------------------------------------------------------------------------------- | | **Default (Auto)** | 512 tok · 0% · Auto | Good everywhere — the safe starting point. | | **FAQ / Q\&A pairs** | 256 tok · 0% · Structural | Small self-contained chunks so each question/answer stays whole. | | **Manual / long-form prose** | 1024 tok · 15% · Recursive | Explanations that span paragraphs — larger chunks with overlap keep them together. | | **Spreadsheet / CSV** | 512 tok · 0% · Structural | Row-aware splitting so rows are never cut in half. | | **Contract / legal** | 768 tok · 20% · Recursive | Generous overlap so clause cross-references survive. | | **Custom** | your values | Any combination you set by hand. | ### Size, overlap and split strategy * **Chunk size (tokens)** — target tokens per chunk (64–4096). Smaller = more precise matches; larger = more context per hit. * **Chunk overlap (%)** — how much consecutive chunks share (0–50%). Overlap keeps an idea that spans a boundary from being cut in half. * **Split strategy** — where cuts land: **Auto** (docling hybrid for documents, paragraph-aware for plain text), **Fixed token** (strict token-count slicing), **Recursive** (paragraph → sentence → token), or **Structural** (Markdown headings, CSV rows, code). ### Metadata per chunk Stamp each chunk with metadata to power filtered retrieval. Set a free-text **Source type tag** (e.g. `policy_doc`, `faq`, `product_spec`) so a query can pre-filter to one document class, and optionally attach **Source path**, **Section title**, **Page number**, **Author** and **Document date**. ### Contextual enrichment Optionally prepend a small header (doc title / section path / a short document summary) to each chunk *before embedding* — Anthropic's "contextual retrieval" recipe, which improves recall on chunks that would otherwise lose their surrounding context. Toggle on **Doc title**, **Section path**, **Doc summary**, or supply a **Custom template** using the placeholders `{doc_title}`, `{section_path}`, `{summary}`, `{chunk}` (default: `{doc_title}` / `{section_path}` / `{chunk}`). ### Review chunks before ingest Turn on **Review chunks before ingest** to make the pipeline pause after chunking, before anything is embedded. The file shows a **Review pending** badge and a **Review chunks** page opens from that file's row, where you can: * see every chunk with its token count, source page/paragraph and headings, next to the converted source markdown; * adjust chunk size, overlap and strategy with live sliders and **re-chunk**, watching the **token-count distribution** histogram for outliers; * **edit**, **split at the cursor**, **merge with the next**, or **delete** individual chunks; * read the estimated **credits to embed** before committing. When it looks right, click **Ingest these chunks** to embed them into the retrieval index — or **Discard preview** to revert the file to a fresh (un-ingested) upload. ### Examples — chunking **Simple — a short FAQ.** Upload your support FAQ as Markdown and leave the **Default (Auto)** preset. Auto splits it into paragraph-aware chunks and it is searchable within seconds — no tuning needed. **Complex — a 200-page manual, reviewed chunk by chunk.** Upload the manual and switch its preset to **Manual / long-form prose** (1024 tokens, 15% overlap, recursive) so multi-paragraph explanations stay together, then turn on **Review chunks before ingest**. When the **Review chunks** page opens from the file's row, scan the token histogram for oversized chunks, split a giant table out of the chunk it landed in, merge two half-sentences, nudge the sliders and re-chunk until the distribution is even — then **Ingest these chunks**. You pay the embedding cost once, on content you have already vetted. ## Per-file knowledge stores Every file row has a collapsible **Knowledge stores** panel showing everything your agent has built *from that document*, with a one-line summary of what exists (chunks · tables · triples · folder) when collapsed. Expand it to inspect and manage each store: * **Vector chunks** — the searchable text passages. **View & edit chunks** opens an inspector where you can review, exclude, split, merge, or edit individual chunks; excluding a chunk removes it from search without deleting the file. * **Structured tables** — any tables detected in the document, kept as queryable grids the agent reads with the `query_table` tool. * **Knowledge graph** — entity–relationship facts extracted from this document (internal files only). **Extract graph** builds one on demand; **Re-extract** rebuilds it (replacing the document's previous graph); **View & edit** opens the graph editor; and you can download it as JSON-LD. * **Document folder** — a self-contained per-document bundle (full text, outline, tables, images) the agent browses with `list_documents` and `get_document`. * **Document profile** — an LLM-written 2–3 sentence summary plus keywords describing what the document is *about*, shown in the agent's document catalog so it routes questions to the right file. **Profile now** / **Re-profile** builds or refreshes it. (Turn on automatic profiling of new uploads on the [Knowledge](/sidebar-menu/knowledge) page.) * **Priority** — a per-document rank boost (**Low**, **Normal**, **High**) that gently tilts ranking when passages from different documents score similarly; it never overrides a clearly better match, and **High**-priority files are exempt from freshness decay. Takes effect once you enable ranking signals on the [Knowledge](/sidebar-menu/knowledge) page. Priority and per-document ontology assignment are staged and applied when you press **Save** in the panel; everything else (profiling, graph extraction, chunk editing, downloads) applies immediately. If a store is missing, the panel shows a hint — re-process the file to (re)build its chunks, tables, and document folder. ## Import from URL and website crawl Beyond uploads, you can pull content straight from the web on the **Import from URL** card: * **Single page or document** — paste a URL and it is fetched, converted and indexed exactly like an uploaded file. Add a **Bearer token** to reach content behind an `Authorization: Bearer …` header. * **Whole-site crawl** — turn on **Include subtree** to crawl every same-site page linked from the URL (sitemap-aware, `robots.txt` respected), up to your plan's page limit. The crawl appears as one expandable entry in the file list rather than one row per page. * **JavaScript rendering** — when available, choose **Off**, **Auto (JS pages)** (render only pages detected as single-page-app shells), or **Always**. Rendering runs each page in a headless browser so SPAs index their real content; rendered pages count against your plan's render limit. * **Automatic re-sync** — set a per-URL cadence (**Off / Daily / Weekly / Monthly**); a background sweep re-fetches on schedule and skips re-indexing when the content is unchanged. You can also **Re-sync** a single URL or **Re-crawl** a whole site on demand. * **Delete** — removing a crawled site's entry deletes every page in that subtree (blobs, vectors and keyword index) in one action. ## Services: ingest from SaaS connectors The **Services** card requires the **Pro** tier or above. The **Services** card ingests records from SaaS tools you have connected (Notion, Google Drive, Zendesk, Salesforce, and so on). First add and authenticate the tool's connector on the **Connectors** page, then on this card: * pick the **Connector** and the **Service**, then **What to ingest** (the object type — tickets, pages, documents…); * narrow the pull with an optional **Filter** and **Extra arguments (JSON)**, plus include/exclude terms and a per-source item cap; * or choose **Custom mapping (any list tool)** to map an arbitrary list endpoint yourself — give the list tool / `operationId`, its arguments, and JSON paths for the item list and each field (id, title, body, URL, updated). GraphQL connectors take a query **document** instead of a tool name. Each source occupies one file slot and ingests up to your plan's item limit. Internal-scope service content is used for the agent's reasoning only, never shown to subscribers. ## External knowledge sources The **External knowledge sources (private)** card connects a vector store or search index you already own, which the agent queries live at chat time alongside your uploaded files — nothing is copied into Agentheya. Supported backends: **AWS Bedrock Knowledge Bases**, **Azure AI Search**, **Google Vertex AI Search**, **OpenAI Vector Stores**, **Pinecone**, **Weaviate**, and a GitHub-hosted **Obsidian** vault. Credentials are encrypted at rest. Because these sources are configured on the *private* page, the agent uses their results as background knowledge but does **not** quote them verbatim to subscribers. To let the agent quote and cite an external source directly, connect it on the [Public Data](/sidebar-menu/data-public) page instead. ## How retrieval works When the agent responds to a query, it uses hybrid retrieval (combining keyword and semantic/vector search) to find the most relevant chunks from the internal data store. Retrieved chunks are included in the agent's context window. To **see exactly what your agent would retrieve** for a given question — which chunks, what scores, what it costs — use the [Retrieval Test](/sidebar-menu/retrieval-test) page. It runs the live retrieval pipeline without spending a chat turn. ## Knowledge graphs Beyond text retrieval, internal documents can be turned into a **knowledge graph** — structured `(subject)→[relationship]→(object)` facts your agent walks to answer connection questions ("who reports to whom?", "what depends on what?"). There are three ways to add graphs on this page: * **Automatically from uploads** — turn on auto-extraction on the [Knowledge](/sidebar-menu/knowledge) page, and uploads of the selected file types build a graph automatically. * **Per document, on demand** — use **Extract graph** in a file's [Knowledge stores](#per-file-knowledge-stores) panel to build (or rebuild) a graph from just that document, whenever you want. * **Direct upload** — the **Knowledge Graphs** card accepts ready-made graphs as JSON triples (a `.json` file of `{subject, predicate, object}` triples, or a `{nodes, edges}` object with richer entity data), a CSV zip (`nodes.csv` + `edges.csv`), or plain text the platform extracts for you. The [Knowledge](/sidebar-menu/knowledge) page controls how those facts are retrieved and cited. ## Ontologies If your domain has a defined vocabulary, import an **ontology** and have knowledge-graph extraction align to it — entities are mapped to your ontology's entity types and relationships to its relationship types, so the resulting graph is consistent and machine-readable. Everything ontology-related lives in the **Ontologies** card on this page: * **Import** ontologies in the formats **RDF/XML** (`.owl`, `.rdf`, `.xml`), **Turtle / N-Triples** (`.ttl`, `.n3`, `.nt`), and **JSON-LD** (`.jsonld`, `.json`). Each import shows how many classes and properties it defines. * **Align document graphs to an ontology** — the master switch (auto-saved) that steers extraction toward your ontology's types and tags each extracted graph with the matched types. This control now lives here on the Internal Data page, next to where ontologies are imported — it is no longer on the Knowledge page. * **Default ontology** — which imported ontology new extractions align to (or *All* to merge every one). Override it per document from that file's **Knowledge stores** panel. * **Strict mode** — drop entities and relationships that don't map onto the ontology, so the graph contains only conformant facts; off keeps them, flagged as un-aligned. Alignment applies to graphs built *after* it is enabled (via auto-extraction or a per-document **Extract graph**); existing graphs are unchanged until re-extracted. Any aligned graph can be downloaded as **JSON-LD** from its Knowledge stores panel — an ontology-linked, machine-readable export you can use in other tools. ### Example — an ontology that answers impact questions Define a small catalog ontology in Turtle so "if X fails, who's affected?" resolves by walking the graph rather than guessing from loose wording: ```turtle theme={null} @prefix : . @prefix rdf: . @prefix rdfs: . :Product a rdfs:Class . :Component a rdfs:Class . :Customer a rdfs:Class . :dependsOn a rdf:Property ; rdfs:domain :Product ; rdfs:range :Component . :affects a rdf:Property ; rdfs:domain :Component ; rdfs:range :Customer . ``` Import this `.ttl`, turn **Align document graphs to an ontology** on, and extract graphs from your product docs. Every document now yields the same `:dependsOn` / `:affects` edges, so when a user asks "if the Titan component fails, which products and customers are affected?" the agent traverses `Component → Product` (reverse `:dependsOn`) and `Component → Customer` (`:affects`) to name them exactly. Enable **strict mode** to keep anything off-schema out of the graph. ## Examples **Simple — an FAQ bot.** Upload your support FAQ as a Markdown file. Once **Loaded**, the agent answers customer questions from it directly. The single most common use of Internal Data. **Medium — a product knowledge base from mixed documents.** Upload a folder of spec sheets (PDF), policy docs (DOCX), and a pricing table (CSV). Each is converted to text; the agent retrieves across all of them. Use [Retrieval Test](/sidebar-menu/retrieval-test) to confirm the right chunks surface for representative questions before going live. **Complex — a graph-backed expert agent.** Upload your internal documentation, turn on knowledge-graph [auto-extraction](/sidebar-menu/knowledge), and set the retrieval strategy to deep traversal. The agent now answers both "what does the warranty say?" (text retrieval) and "if component X fails, which products and customers are affected?" (graph traversal) from the same uploaded files — with answers cited back to the source passages. ## File management * **Download** — use the download icon to fetch the original file. * **Delete** — use the trash icon to remove a file. After saving, the file is removed from the search index and is no longer retrievable. * **Re-upload** — the same filename cannot be re-uploaded. Delete the existing file first if you want to replace it. ## Limits File size and total storage limits depend on your subscription tier. The per-file maximum is enforced both client-side and server-side. ## Manage via the Management MCP This page can be managed programmatically through the [Management MCP](/manage-mcp). | | | | --------------- | ------------------------------------------------------------------------------------------------------- | | Endpoint | `https://manage.agentheya.com/mcp` | | Upload | `file.upload` with `scope: "internal"` | | List | `file.list` with `scope: "internal"` | | Delete | `file.delete` with `scope: "internal"` | | Chunking config | `config.set` section `data_internal` | | Schema (HTTP) | [`GET /api/manage-mcp/schema/data_internal`](https://agentheya.com/api/manage-mcp/schema/data_internal) | Example upload: ```json theme={null} { "tool": "file.upload", "owner_id": "", "agent_id": "", "scope": "internal", "filename": "faq.md", "content": "# FAQ\n\n..." } ``` Per-plan size/format limits apply. Ingestion runs asynchronously after upload.