data_internal

Private knowledge the agent retrieves from but never exposes verbatim: uploaded files, indexed URLs, and optional external vector stores.

data_internal — Internal data

Private knowledge the agent retrieves from but never exposes verbatim: uploaded files, indexed URLs, and optional external vector stores.

Section key data_internal · Shape object · Tier All plans
Edit in the dashboard: Internal Data page
Related: external_stores backends

Fields

Field	Required	Type	Allowed values / constraints	Default	Description
files	no	array of object	—	—	Uploaded documents processed for retrieval-augmented generation. Item fields below.
urls	no	array of object	—	—	URL sources fetched and indexed for RAG. Item fields below.
external_stores	no	array of object	—	[]	External knowledge sources the agent should query at chat time alongside the agent’s built-in knowledge base. Each entry is a connection to YOUR existing store — a Pinecone or Weaviate cluster, Bedrock KB, Azure AI Search, Vertex AI Search, OpenAI Vector Store, or a GitHub-hosted Obsidian vault — Agentheya does not ingest into it, only queries. Hits are merged with built-in retrieval results. Credentials (api_key fields) are encrypted at rest. See external_stores backends for the full sub-schema.
service_sources	no	array of object	Pro+	—	Configured SaaS service data sources (Services card). Each pulls documents/records through a connector added on the /connectors page and ingests them into this scope’s retrieval index. Sync-state fields (last_synced_at, last_sync_status, last_sync_error, item_count) are server-stamped and never trusted from clients. Item fields below.

files items

Uploaded documents processed for retrieval-augmented generation.

Field	Required	Type	Allowed values / constraints	Default	Description
name	yes	string	—	—	Original file name.
path	yes	string	—	—	Stored file path relative to the agent’s files directory.
doc_id	no	string	—	—	Stable short identifier for this ingested object (e.g. ‘doc_k7m2p9q4’). Minted once when the entry first enters the manifest and preserved across renames, re-syncs and re-ingests. Shown on the data page and on the matching Activity ingest row.
size	yes	string	—	—	Human-readable file size (e.g. ‘1.2 MB’).
chunk_count	no	integer	—	—	Number of chunks produced by the ingest pipeline (set after processing).
total_tokens	no	integer	—	—	Total token count across all chunks (set after processing).
chunking_config	no	object	—	—	How the file is split for retrieval. Item fields below.
retrieval_enabled	no	boolean	—	—	When false, the file is kept on disk and indexed but excluded from retrieval (keyword + vector). Default true.
excluded_chunks	no	array of integer	—	—	Chunk indexes excluded from retrieval at query time (set from the chunk inspector). Chunks stay indexed so re-enabling is instant.
content_hash	no	string	—	—	SHA-256 hex of the stored file content. Set on upload / URL fetch; used for change detection on URL re-sync.
source_url	no	string	—	—	Set when this file was imported from a URL; enables re-sync.
url_sync	no	enum	one of manual, daily, weekly, monthly	—	Re-sync cadence for URL-imported files. ‘daily’ / ‘weekly’ / ‘monthly’ automatically re-sync once the period since last_synced_at has passed; ‘manual’ re-syncs only on request.
last_synced_at	no	string	—	—	ISO timestamp of the last successful URL fetch (URL-imported files only).
render_mode	no	enum	one of auto, always	—	JS render mode this URL was imported with (web-render microservice); re-syncs re-apply it so a rendered page never reverts to its raw shell.
site_group	no	string	—	—	Site host (www-folded) this page was imported under by a whole-website crawl. Pages sharing a site_group display as ONE subtree entry on the data page and count as ONE file toward the per-plan file-count limits.
import_root_url	no	string	—	—	Start URL of the whole-website crawl that imported this page — used by the subtree entry’s re-crawl action.
align_ontology	no	string	max len 200	—	Per-document ontology alignment override applied when this document’s knowledge graph is (re-)extracted. Empty/unset = use the agent’s default ontology; ‘all’ = every imported ontology; ‘off’ = do not align this document even when alignment is on; otherwise the stored_name of a specific imported ontology.
doc_profile	no	object	—	—	LLM-generated knowledge profile written by the ingest pipeline (not owner-editable): what the document is about + topic keywords. Feeds the agent’s document catalog so it can route to the right document. Item fields below.
rank_boost	no	enum	one of low, high	—	Owner-declared retrieval priority applied as a post-fusion score multiplier. ‘normal’ is stored by omission.
service_source_id	no	string	—	—	Id of the service source (service_sources[].id) that ingested this file. Files sharing a service_source_id render as one expandable Services entry and count as ONE file slot.
service_item_key	no	string	—	—	Stable upstream identity ‘<source_id>:<item_id>’ — the re-sync upsert key (the way source_url keys URL imports).
service_updated_at	no	string	—	—	Upstream last-modified timestamp of the item at the last sync.

urls items

URL sources fetched and indexed for RAG.

Field	Required	Type	Allowed values / constraints	Default	Description
url	yes	string	—	—	The URL to fetch.
bearer_token	no	string	—	—	Optional bearer token sent when fetching the URL.

service_sources items

Configured SaaS service data sources (Services card). Each pulls documents/records through a connector added on the /connectors page and ingests them into this scope’s retrieval index. Sync-state fields (last_synced_at, last_sync_status, last_sync_error, item_count) are server-stamped and never trusted from clients.

Field	Required	Type	Allowed values / constraints	Default	Description
id	yes	string	pattern ^svc_[a-z0-9]+$	—	Stable source id; ingested files reference it via service_source_id.
name	yes	string	max len 120	—	Display name shown on the data page.
connector_kind	yes	enum	one of composio, mcp, openapi, graphql, s3, native	—	Which connector mechanism this source pulls through.
connector_ref	yes	string	max len 200	—	Composio record id (stringified number) or connector name from the connectors section.
service	yes	string	max len 64	—	Recipe key in config/service-ingest-catalog.json (e.g. ‘googledrive’, ‘jira’), or ‘custom’.
object_type	no	string	max len 64	—	Object-type key within the recipe (e.g. ‘files’, ‘issues’).
query	no	string	max len 2000	—	Recipe-specific listing filter (Drive query, JQL, SOQL WHERE, search text…).
extra_args_json	no	string	max len 4000	—	JSON object string merged into the list tool’s arguments — for recipes whose tools need per-source identifiers (Sentry org slug, Bitbucket workspace/repo, BambooHR subdomain…). GraphQL: becomes the query variables.
include_terms	no	string	max len 1000	—	Comma-separated terms — when set, an item’s title must contain one to be ingested.
exclude_terms	no	string	max len 1000	—	Comma-separated terms — items whose title contains one are skipped.
max_items	no	integer	min 1	—	Per-source item cap; clamped to the plan’s service_import_item_limits.
sync	yes	enum	one of manual, daily, weekly, monthly	—	Automatic re-sync cadence; next sync is due last_synced_at + period.
chunking_config	no	object	—	—	Same shape as files[].chunking_config; inherited by every file this source ingests.
custom	no	object	—	—	Owner-authored field mapping when service === ‘custom’ (build-your-own connector). Item fields below.
last_synced_at	no	string	—	—	Server-stamped ISO time of the last completed sync.
last_sync_status	no	enum	one of ok, error, “	—
last_sync_error	no	string	max len 2000	—
item_count	no	integer	—	—	Files currently held by this source.
s3	no	object	—	—	Direct S3 / S3-compatible bucket config (connector_kind ‘s3’). secret_access_key is encrypted at rest and read back as ‘REDACTED’. Item fields below.

files.chunking_config fields

How the file is split for retrieval.

Field	Required	Type	Allowed values / constraints	Default	Description
chunk_size	no	integer	—	—	Target chunk size in tokens. Default 512.
chunk_overlap_pct	no	integer	—	—	Overlap between adjacent chunks as a percentage of chunk_size. Default 0.
split_strategy	no	enum	one of auto, fixed_token, recursive, semantic, structural	—	Chunking strategy. ‘auto’ is recommended.
metadata	no	object	—	—	Source metadata stamped on each chunk. Item fields below.
contextual_enrichment	no	object	—	—	Prepend auto-generated context to each chunk to improve retrieval. Item fields below.
preview_before_ingest	no	boolean	—	—	When true, the file is converted and chunked but PAUSES before indexing so the owner can preview and optionally edit chunks on the preview page, then clicks ‘Ingest these chunks’ to finish. Default false.

files.doc_profile fields

LLM-generated knowledge profile written by the ingest pipeline (not owner-editable): what the document is about + topic keywords. Feeds the agent’s document catalog so it can route to the right document.

Field	Required	Type	Allowed values / constraints	Default	Description
summary	no	string	max len 500	—	2–3 sentence summary of what the document is about and what kinds of questions it answers.
keywords	no	array of string	max 8 items	—	Up to 8 topic keywords / entity names, most distinctive first.
profiled_at	no	string	—	—	ISO timestamp of the profiling run.

service_sources.custom fields

Owner-authored field mapping when service === ‘custom’ (build-your-own connector).

Field	Required	Type	Allowed values / constraints	Default	Description
list_tool	yes	string	max len 4000	—	Connector tool (Composio/MCP name, OpenAPI operationId) that lists items — or, for GraphQL connectors, the full query document.
list_args_json	no	string	max len 4000	—	JSON object string of extra arguments for the list tool.
items_path	no	string	max len 200	—	Dot-path to the items array in the result; empty = auto-discover.
id_path	no	string	max len 200	—
title_path	no	string	max len 200	—
body_path	no	string	max len 200	—
url_path	no	string	max len 200	—
updated_path	no	string	max len 200	—

service_sources.s3 fields

Direct S3 / S3-compatible bucket config (connector_kind ‘s3’). secret_access_key is encrypted at rest and read back as ‘REDACTED’.

Field	Required	Type	Allowed values / constraints	Default	Description
bucket	yes	string	max len 255	—
region	no	string	max len 64	—
endpoint	no	string	max len 400	—	Custom https endpoint for R2 / Spaces / MinIO; empty = AWS.
prefix	no	string	max len 512	—	Only objects under this key prefix are listed.
access_key_id	no	string	max len 256	—
secret_access_key	no	string	max len 2048	—

files.chunking_config.metadata fields

Source metadata stamped on each chunk.

Field	Required	Type	Allowed values / constraints	Default	Description
source_type	no	string	—	—	Free-text tag (max 64 chars) for filtered retrieval, e.g. ‘policy_doc’, ‘faq’, ‘product_spec’. Stamped on every chunk’s metadata, enabling queries to pre-filter by source_type in retrieval.
include_source	no	boolean	—	—	Boolean. When true, attaches the file’s source path or URL to each chunk’s metadata as the ‘source’ field for citations and retrieval tracing.
include_section_title	no	boolean	—	—	Boolean. When true, stamps each chunk’s metadata with ‘section_title’ (the heading/title from the document’s structure) and ‘section_path’ (the hierarchy of headings) if available.
include_page_number	no	boolean	—	—	Boolean. When true, includes page number in each chunk’s metadata if the source format supports it (e.g. PDF, DOCX). Currently unavailable for most formats.
include_author	no	boolean	—	—	Boolean. When true and author field is set, attaches the author name to each chunk’s metadata for provenance tracking.
author	no	string	—	—	String (max 128 chars, e.g. ‘Legal team’). Stamped on chunks when include_author is true. Identifies the document’s author or responsible party.
include_date	no	boolean	—	—	Boolean. When true and doc_date field is set, attaches the document date to each chunk’s metadata for temporal filtering and versioning.
doc_date	no	string	—	—	String (max 64 chars: YYYY-MM-DD format or free text). Stamped on chunks when include_date is true. Captures when the document was authored or last updated.

files.chunking_config.contextual_enrichment fields

Prepend auto-generated context to each chunk to improve retrieval.

Field	Required	Type	Allowed values / constraints	Default	Description
enabled	no	boolean	—	—	Boolean. When true, each chunk is prefixed with auto-generated context (doc title / section path) before embedding, which improves retrieval quality.
include_doc_title	no	boolean	—	—	Boolean. When true (and enrichment enabled), prepends the document’s title to each chunk for retrieval context.
include_section_path	no	boolean	—	—	Boolean. When true (and enrichment enabled), prepends the section/heading hierarchy (e.g. ‘Chapter 2 › Section 2.3’) to each chunk for navigation context.
include_summary	no	boolean	—	—	Boolean. When true (and enrichment enabled), prepends a short extractive document lead summary (the doc’s opening sentences) to each chunk via the {summary} placeholder. Deterministic — no LLM call. Default false.
custom_template	no	string	—	—	String (max 2000 chars). Optional Jinja-like template controlling how context is formatted: {doc_title}, {section_path}, {summary}, {chunk}. Defaults to ‘{doc_title}\n{section_path}\n\n{chunk}’ when empty.