Internal Data

Upload private files that your agent uses for retrieval-augmented generation.

Field reference: data_internal — every field, type, default, and tier for this page.

Internal Data is private knowledge your agent retrieves to inform its answers. Files you add here are indexed and made searchable. When a user asks a question, the agent retrieves relevant chunks and uses them as context — this is retrieval-augmented generation (RAG).

Internal data is never shared with subscribers as downloadable files. It is used only to inform the agent’s responses.

Adding files

1. Open your agent and go to Internal Data in the sidebar.
2. Drag and drop files onto the drop zone, or click it to browse.
3. The file uploads and starts indexing automatically — changes on this page save on their own (there is no Save button).

Supported file types: text formats (Markdown, JSON, CSV, plain text) plus documents that can be converted to text — PDF, DOCX, XLSX, HTML and similar are automatically converted to text and structure before indexing. Executable files and other binaries are rejected.

The maximum file size is set by your subscription plan and enforced on upload (both client-side and server-side).

Indexing status

After saving, each file shows a status badge as the background ingest pipeline processes it:

Badge	Meaning
Uploading…	File picked locally, upload in progress.
Queued / Queued (next) / Queued (N ahead)	Waiting in the ingest queue.
Converting…	The source file is being converted to a searchable text format.
Review pending	Chunking paused for your review (see Review chunks before ingest) — the file waits until you ingest or discard the preview.
Processing…	Chunks are being embedded and written to the retrieval store.
Loaded	Indexed and searchable by the agent.
Error: …	Ingest failed; hover the badge for details.

Once a file is Loaded, you can click the info icon to view file stats (chunk count and total tokens).

Chunking configuration

Before a file is indexed it is split into chunks — the passages retrieval actually searches and returns. Each file carries its own chunking config, opened from the Chunking & metadata control on the file’s row. Defaults are sensible for most documents; tune only when retrieval quality is off. Changes take effect the next time the file is (re-)processed.

Presets

Start from a preset, then fine-tune if needed. A preset only sets the three splitting knobs (size, overlap, strategy) — metadata and enrichment settings survive a preset switch.

Preset	Size · overlap · strategy	Use for
Default (Auto)	512 tok · 0% · Auto	Good everywhere — the safe starting point.
FAQ / Q&A pairs	256 tok · 0% · Structural	Small self-contained chunks so each question/answer stays whole.
Manual / long-form prose	1024 tok · 15% · Recursive	Explanations that span paragraphs — larger chunks with overlap keep them together.
Spreadsheet / CSV	512 tok · 0% · Structural	Row-aware splitting so rows are never cut in half.
Contract / legal	768 tok · 20% · Recursive	Generous overlap so clause cross-references survive.
Custom	your values	Any combination you set by hand.

Size, overlap and split strategy

• Chunk size (tokens) — target tokens per chunk (64–4096). Smaller = more precise matches; larger = more context per hit.
• Chunk overlap (%) — how much consecutive chunks share (0–50%). Overlap keeps an idea that spans a boundary from being cut in half.
• Split strategy — where cuts land: Auto (docling hybrid for documents, paragraph-aware for plain text), Fixed token (strict token-count slicing), Recursive (paragraph → sentence → token), or Structural (Markdown headings, CSV rows, code).

Metadata per chunk

Stamp each chunk with metadata to power filtered retrieval. Set a free-text Source type tag (e.g. policy_doc, faq, product_spec) so a query can pre-filter to one document class, and optionally attach Source path, Section title, Page number, Author and Document date.

Contextual enrichment

Optionally prepend a small header (doc title / section path / a short document summary) to each chunk before embedding — Anthropic’s “contextual retrieval” recipe, which improves recall on chunks that would otherwise lose their surrounding context. Toggle on Doc title, Section path, Doc summary, or supply a Custom template using the placeholders {doc_title}, {section_path}, {summary}, {chunk} (default: {doc_title} / {section_path} / {chunk}).

Review chunks before ingest

Turn on Review chunks before ingest to make the pipeline pause after chunking, before anything is embedded. The file shows a Review pending badge and a Review chunks page opens from that file’s row, where you can:

• see every chunk with its token count, source page/paragraph and headings, next to the converted source markdown;
• adjust chunk size, overlap and strategy with live sliders and re-chunk, watching the token-count distribution histogram for outliers;
• edit, split at the cursor, merge with the next, or delete individual chunks;
• read the estimated credits to embed before committing.

When it looks right, click Ingest these chunks to embed them into the retrieval index — or Discard preview to revert the file to a fresh (un-ingested) upload.

Examples — chunking

Simple — a short FAQ. Upload your support FAQ as Markdown and leave the Default (Auto) preset. Auto splits it into paragraph-aware chunks and it is searchable within seconds — no tuning needed.

Complex — a 200-page manual, reviewed chunk by chunk. Upload the manual and switch its preset to Manual / long-form prose (1024 tokens, 15% overlap, recursive) so multi-paragraph explanations stay together, then turn on Review chunks before ingest. When the Review chunks page opens from the file’s row, scan the token histogram for oversized chunks, split a giant table out of the chunk it landed in, merge two half-sentences, nudge the sliders and re-chunk until the distribution is even — then Ingest these chunks. You pay the embedding cost once, on content you have already vetted.

Per-file knowledge stores

Every file row has a collapsible Knowledge stores panel showing everything your agent has built from that document, with a one-line summary of what exists (chunks · tables · triples · folder) when collapsed. Expand it to inspect and manage each store:

• Vector chunks — the searchable text passages. View & edit chunks opens an inspector where you can review, exclude, split, merge, or edit individual chunks; excluding a chunk removes it from search without deleting the file.
• Structured tables — any tables detected in the document, kept as queryable grids the agent reads with the query_table tool.
• Knowledge graph — entity–relationship facts extracted from this document (internal files only). Extract graph builds one on demand; Re-extract rebuilds it (replacing the document’s previous graph); View & edit opens the graph editor; and you can download it as JSON-LD.
• Document folder — a self-contained per-document bundle (full text, outline, tables, images) the agent browses with list_documents and get_document.
• Document profile — an LLM-written 2–3 sentence summary plus keywords describing what the document is about, shown in the agent’s document catalog so it routes questions to the right file. Profile now / Re-profile builds or refreshes it. (Turn on automatic profiling of new uploads on the Knowledge page.)
• Priority — a per-document rank boost (Low, Normal, High) that gently tilts ranking when passages from different documents score similarly; it never overrides a clearly better match, and High-priority files are exempt from freshness decay. Takes effect once you enable ranking signals on the Knowledge page.

Priority and per-document ontology assignment are staged and applied when you press Save in the panel; everything else (profiling, graph extraction, chunk editing, downloads) applies immediately. If a store is missing, the panel shows a hint — re-process the file to (re)build its chunks, tables, and document folder.

Import from URL and website crawl

Beyond uploads, you can pull content straight from the web on the Import from URL card:

• Single page or document — paste a URL and it is fetched, converted and indexed exactly like an uploaded file. Add a Bearer token to reach content behind an Authorization: Bearer … header.
• Whole-site crawl — turn on Include subtree to crawl every same-site page linked from the URL (sitemap-aware, robots.txt respected), up to your plan’s page limit. The crawl appears as one expandable entry in the file list rather than one row per page.
• JavaScript rendering — when available, choose Off, Auto (JS pages) (render only pages detected as single-page-app shells), or Always. Rendering runs each page in a headless browser so SPAs index their real content; rendered pages count against your plan’s render limit.
• Automatic re-sync — set a per-URL cadence (Off / Daily / Weekly / Monthly); a background sweep re-fetches on schedule and skips re-indexing when the content is unchanged. You can also Re-sync a single URL or Re-crawl a whole site on demand.
• Delete — removing a crawled site’s entry deletes every page in that subtree (blobs, vectors and keyword index) in one action.

Services: ingest from SaaS connectors

The Services card requires the Pro tier or above.

The Services card ingests records from SaaS tools you have connected (Notion, Google Drive, Zendesk, Salesforce, and so on). First add and authenticate the tool’s connector on the Connectors page, then on this card:

• pick the Connector and the Service, then What to ingest (the object type — tickets, pages, documents…);
• narrow the pull with an optional Filter and Extra arguments (JSON), plus include/exclude terms and a per-source item cap;
• or choose Custom mapping (any list tool) to map an arbitrary list endpoint yourself — give the list tool / operationId, its arguments, and JSON paths for the item list and each field (id, title, body, URL, updated). GraphQL connectors take a query document instead of a tool name.

Each source occupies one file slot and ingests up to your plan’s item limit. Internal-scope service content is used for the agent’s reasoning only, never shown to subscribers.

External knowledge sources

The External knowledge sources (private) card connects a vector store or search index you already own, which the agent queries live at chat time alongside your uploaded files — nothing is copied into Agentheya. Supported backends: AWS Bedrock Knowledge Bases, Azure AI Search, Google Vertex AI Search, OpenAI Vector Stores, Pinecone, Weaviate, and a GitHub-hosted Obsidian vault. Credentials are encrypted at rest.

Because these sources are configured on the private page, the agent uses their results as background knowledge but does not quote them verbatim to subscribers. To let the agent quote and cite an external source directly, connect it on the Public Data page instead.

How retrieval works

When the agent responds to a query, it uses hybrid retrieval (combining keyword and semantic/vector search) to find the most relevant chunks from the internal data store. Retrieved chunks are included in the agent’s context window.

To see exactly what your agent would retrieve for a given question — which chunks, what scores, what it costs — use the Retrieval Test page. It runs the live retrieval pipeline without spending a chat turn.

Knowledge graphs

Beyond text retrieval, internal documents can be turned into a knowledge graph — structured (subject)→[relationship]→(object) facts your agent walks to answer connection questions (“who reports to whom?”, “what depends on what?”). There are three ways to add graphs on this page:

• Automatically from uploads — turn on auto-extraction on the Knowledge page, and uploads of the selected file types build a graph automatically.
• Per document, on demand — use Extract graph in a file’s Knowledge stores panel to build (or rebuild) a graph from just that document, whenever you want.
• Direct upload — the Knowledge Graphs card accepts ready-made graphs as JSON triples (a .json file of {subject, predicate, object} triples, or a {nodes, edges} object with richer entity data), a CSV zip (nodes.csv + edges.csv), or plain text the platform extracts for you.

The Knowledge page controls how those facts are retrieved and cited.

Ontologies

If your domain has a defined vocabulary, import an ontology and have knowledge-graph extraction align to it — entities are mapped to your ontology’s entity types and relationships to its relationship types, so the resulting graph is consistent and machine-readable.

Everything ontology-related lives in the Ontologies card on this page:

• Import ontologies in the formats RDF/XML (.owl, .rdf, .xml), Turtle / N-Triples (.ttl, .n3, .nt), and JSON-LD (.jsonld, .json). Each import shows how many classes and properties it defines.
• Align document graphs to an ontology — the master switch (auto-saved) that steers extraction toward your ontology’s types and tags each extracted graph with the matched types. This control now lives here on the Internal Data page, next to where ontologies are imported — it is no longer on the Knowledge page.
• Default ontology — which imported ontology new extractions align to (or All to merge every one). Override it per document from that file’s Knowledge stores panel.
• Strict mode — drop entities and relationships that don’t map onto the ontology, so the graph contains only conformant facts; off keeps them, flagged as un-aligned.

Alignment applies to graphs built after it is enabled (via auto-extraction or a per-document Extract graph); existing graphs are unchanged until re-extracted. Any aligned graph can be downloaded as JSON-LD from its Knowledge stores panel — an ontology-linked, machine-readable export you can use in other tools.

Example — an ontology that answers impact questions

Define a small catalog ontology in Turtle so “if X fails, who’s affected?” resolves by walking the graph rather than guessing from loose wording:

@prefix :     <http://example.com/catalog#> .
@prefix rdf:  <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .

:Product   a rdfs:Class .
:Component a rdfs:Class .
:Customer  a rdfs:Class .

:dependsOn a rdf:Property ; rdfs:domain :Product   ; rdfs:range :Component .
:affects   a rdf:Property ; rdfs:domain :Component ; rdfs:range :Customer .

Import this .ttl, turn Align document graphs to an ontology on, and extract graphs from your product docs. Every document now yields the same :dependsOn / :affects edges, so when a user asks “if the Titan component fails, which products and customers are affected?” the agent traverses Component → Product (reverse :dependsOn) and Component → Customer (:affects) to name them exactly. Enable strict mode to keep anything off-schema out of the graph.

Examples

Simple — an FAQ bot. Upload your support FAQ as a Markdown file. Once Loaded, the agent answers customer questions from it directly. The single most common use of Internal Data.

Medium — a product knowledge base from mixed documents. Upload a folder of spec sheets (PDF), policy docs (DOCX), and a pricing table (CSV). Each is converted to text; the agent retrieves across all of them. Use Retrieval Test to confirm the right chunks surface for representative questions before going live.

Complex — a graph-backed expert agent. Upload your internal documentation, turn on knowledge-graph auto-extraction, and set the retrieval strategy to deep traversal. The agent now answers both “what does the warranty say?” (text retrieval) and “if component X fails, which products and customers are affected?” (graph traversal) from the same uploaded files — with answers cited back to the source passages.

File management

• Download — use the download icon to fetch the original file.
• Delete — use the trash icon to remove a file. After saving, the file is removed from the search index and is no longer retrievable.
• Re-upload — the same filename cannot be re-uploaded. Delete the existing file first if you want to replace it.

Limits

File size and total storage limits depend on your subscription tier. The per-file maximum is enforced both client-side and server-side.

Manage via the Management MCP

This page can be managed programmatically through the Management MCP.

Endpoint	https://manage.agentheya.com/mcp
Upload	file.upload with scope: "internal"
List	file.list with scope: "internal"
Delete	file.delete with scope: "internal"
Chunking config	config.set section data_internal
Schema (HTTP)	GET /api/manage-mcp/schema/data_internal

Example upload:

{
  "tool": "file.upload",
  "owner_id": "<your owner_id>",
  "agent_id": "<your agent_id>",
  "scope": "internal",
  "filename": "faq.md",
  "content": "# FAQ\n\n..."
}

Per-plan size/format limits apply. Ingestion runs asynchronously after upload.