data_public

Knowledge sources the agent may quote and surface to power-users — public files and URLs.

data_public — Public data

Knowledge sources the agent may quote and surface to power-users — public files and URLs.

Section key data_public · Shape object · Tier All plans
Edit in the dashboard: Public Data page

Fields

Field	Required	Type	Allowed values / constraints	Default	Description
files	no	array of object	—	—	Uploaded documents processed for retrieval-augmented generation. Item fields below.
urls	no	array of object	—	—	Array of URL sources to fetch and index for RAG. Each URL is polled, converted to chunks, and indexed into the same hybrid keyword + semantic retrieval index as public files. Supports optional bearer-token auth for protected endpoints. Item fields below.
external_stores	no	array of object	—	[]	External knowledge sources the agent should query at chat time alongside the agent’s built-in knowledge base. Each entry is a connection to YOUR existing store — a Pinecone or Weaviate cluster, Bedrock KB, Azure AI Search, Vertex AI Search, OpenAI Vector Store, or a GitHub-hosted Obsidian vault — Agentheya does not ingest into it, only queries. Hits are merged with built-in retrieval results. Credentials (api_key fields) are encrypted at rest. Item fields below.
service_sources	no	array of object	Pro+	—	Configured SaaS service data sources (Services card). Each pulls documents/records through a connector added on the /connectors page and ingests them into this scope’s retrieval index. Sync-state fields (last_synced_at, last_sync_status, last_sync_error, item_count) are server-stamped and never trusted from clients. Item fields below.

files items

Uploaded documents processed for retrieval-augmented generation.

Field	Required	Type	Allowed values / constraints	Default	Description
name	yes	string	—	—	Original filename as uploaded by the agent owner. Used to identify the file in presentation tools (present_public_file, show_file_excerpt) and in the agent’s chat context.
path	yes	string	—	—	File reference set during upload; used to locate the file for retrieval and presentation.
doc_id	no	string	—	—	Stable short identifier for this ingested object (e.g. ‘doc_k7m2p9q4’). Minted once when the entry first enters the manifest and preserved across renames, re-syncs and re-ingests. Shown on the data page and on the matching Activity ingest row.
size	yes	string	—	—	Human-readable file size (e.g. ‘1.2 MB’). Displayed in the dashboard and returned by presentation tools for power-user context.
chunk_count	no	integer	—	—	Number of chunks produced after the ingest pipeline processes the file. Set post-processing; indicates how the file was segmented for retrieval.
total_tokens	no	integer	—	—	Total token count across all chunks in the file. Set post-processing; used to estimate retrieval cost and provide usage metrics.
chunking_config	no	object	—	—	Configuration object controlling how the file is split into chunks for RAG retrieval. Passed to the ingest pipeline; controls chunk size, overlap, strategy, metadata stamping, and contextual enrichment. Item fields below.
retrieval_enabled	no	boolean	—	—	When false, the file is kept on disk and indexed but excluded from retrieval (keyword + vector). Default true.
excluded_chunks	no	array of integer	—	—	Chunk indexes excluded from retrieval at query time (set from the chunk inspector). Chunks stay indexed so re-enabling is instant.
content_hash	no	string	—	—	SHA-256 hex of the stored file content. Set on upload / URL fetch; used for change detection on URL re-sync.
source_url	no	string	—	—	Set when this file was imported from a URL; enables re-sync.
url_sync	no	enum	one of manual, daily, weekly, monthly	—	Re-sync cadence for URL-imported files. ‘daily’ / ‘weekly’ / ‘monthly’ automatically re-sync once the period since last_synced_at has passed; ‘manual’ re-syncs only on request.
last_synced_at	no	string	—	—	ISO timestamp of the last successful URL fetch (URL-imported files only).
render_mode	no	enum	one of auto, always	—	JS render mode this URL was imported with (web-render microservice); re-syncs re-apply it so a rendered page never reverts to its raw shell.
site_group	no	string	—	—	Site host (www-folded) this page was imported under by a whole-website crawl. Pages sharing a site_group display as ONE subtree entry on the data page and count as ONE file toward the per-plan file-count limits.
import_root_url	no	string	—	—	Start URL of the whole-website crawl that imported this page — used by the subtree entry’s re-crawl action.
doc_profile	no	object	—	—	LLM-generated knowledge profile written by the ingest pipeline (not owner-editable): what the document is about + topic keywords. Feeds the agent’s document catalog so it can route to the right document. Item fields below.
rank_boost	no	enum	one of low, high	—	Owner-declared retrieval priority applied as a post-fusion score multiplier. ‘normal’ is stored by omission.
service_source_id	no	string	—	—	Id of the service source (service_sources[].id) that ingested this file. Files sharing a service_source_id render as one expandable Services entry and count as ONE file slot.
service_item_key	no	string	—	—	Stable upstream identity ‘<source_id>:<item_id>’ — the re-sync upsert key (the way source_url keys URL imports).
service_updated_at	no	string	—	—	Upstream last-modified timestamp of the item at the last sync.

urls items

Array of URL sources to fetch and index for RAG. Each URL is polled, converted to chunks, and indexed into the same hybrid keyword + semantic retrieval index as public files. Supports optional bearer-token auth for protected endpoints.

Field	Required	Type	Allowed values / constraints	Default	Description
url	yes	string	—	—	The HTTP(s) URL to fetch. The platform fetches the content, splits it according to chunking_config defaults, and indexes it for agent retrieval. URL-imported files support automatic re-sync via the url_sync field on the corresponding file entry.
bearer_token	no	string	—	—	Optional bearer token sent in the Authorization header when fetching the URL. Used for protected endpoints; if provided, it is sent as ‘Authorization: Bearer <bearer_token>’. Leave empty for public URLs.

external_stores items

External knowledge sources the agent should query at chat time alongside the agent’s built-in knowledge base. Each entry is a connection to YOUR existing store — a Pinecone or Weaviate cluster, Bedrock KB, Azure AI Search, Vertex AI Search, OpenAI Vector Store, or a GitHub-hosted Obsidian vault — Agentheya does not ingest into it, only queries. Hits are merged with built-in retrieval results. Credentials (api_key fields) are encrypted at rest.

Field	Required	Type	Allowed values / constraints	Default	Description
id	yes	string	min len 1	—	Stable kebab-case id, unique within the section’s external_stores array.
kind	yes	enum	one of pinecone, weaviate, bedrock_retrieve, bedrock_retrieve_and_generate, azure_ai_search, vertex_search, openai_vector_store, obsidian	—	Which knowledge-source backend this entry points at. Raw vector stores (pinecone/weaviate) need query-embedding config; the managed providers (bedrock_*/azure_ai_search/vertex_search/openai_vector_store) embed the query themselves; obsidian is a git-hosted Obsidian vault searched lexically (no embedding config).
auto_retrieve	no	boolean	—	true	When on (default), this source is searched automatically on every turn and its hits are injected into context. Turn off to only expose it via the search_knowledge_base tool.
expose_as_tool	no	boolean	—	false	When on, this source is reachable by the LLM-callable search_knowledge_base tool so the model can search it on demand. Public sources are also readable in full via read_knowledge_document, so the agent can quote entire excerpts verbatim.
name	yes	string	min len 1	—	Human label shown in dashboards and retrieval traces.
enabled	no	boolean	—	true	Off-switch. Disabled stores are skipped at chat time.
collapsed	no	boolean	—	false	Dashboard editor UI state.
top_k	no	integer	min 1; max 50	8	Max hits to fetch per query.
text_field	no	string	—	—	Metadata/property field on each record that holds the original chunk text. Surfaced in LLM context.
title_field	no	string	—	—	Optional metadata field with a short human title for citations.
source_field	no	string	—	—	Optional metadata field with a URL or origin path for citations.
filter	no	object	—	—	Optional metadata/where filter passed verbatim to the store at query time. Provider-specific shape (Pinecone metadata filter / Weaviate where filter).
pinecone_host	no	string	—	—	Pinecone index host (e.g. ‘my-index-abc.svc.aped-us-east-1-fkj4.pinecone.io’). Copy from the Pinecone console — works for both serverless and pod-based indexes.
pinecone_index	no	string	—	—	Pinecone index name (e.g. ‘docs’).
pinecone_namespace	no	string	—	—	Optional Pinecone namespace. Empty = default namespace.
pinecone_api_key	no	string	—	—	Pinecone API key. Stored encrypted at rest; UI reads return ‘REDACTED’.
weaviate_url	no	string	—	—	Weaviate cluster base URL (e.g. ‘https://my-cluster.weaviate.network’). Trailing slash optional.
weaviate_class	no	string	—	—	Weaviate collection / class name to query.
weaviate_api_key	no	string	—	—	Weaviate API key. Empty allowed for clusters without auth. Stored encrypted at rest.
weaviate_use_built_in_vectorizer	no	boolean	—	false	When true, queries use the collection’s configured vectorizer (nearText). When false, Agentheya embeds the query itself and passes a nearVector — requires embedding_* fields.
weaviate_tenant	no	string	—	—	Optional tenant name for Weaviate multi-tenant collections. When set, every query is scoped via Get { Class(tenant: "…") { … } }. Leave empty for single-tenant collections.
weaviate_query_mode	no	enum	one of near, hybrid	near	Search operator. ‘near’ = nearText|nearVector (pure vector similarity). ‘hybrid’ = hybrid { query alpha vector? } which combines keyword recall with vector similarity — almost always retrieves better than either alone.
weaviate_hybrid_alpha	no	number	min 0; max 1	0.5	Blend factor when weaviate_query_mode=‘hybrid’. 0 = keyword only, 1 = vector only, 0.5 = balanced (Weaviate default). Ignored when mode=‘near’.
embedding_provider	no	enum	one of openai, voyage, cohere, gemini, mistral, jina, azure-openai, huggingface, custom	—	Provider used to embed the QUERY before sending it to the store. MUST match the provider+model+dimensions used at ingest — vectors from different models live in incompatible spaces. Use ‘custom’ (HTTP escape hatch) for any provider not in the native list (Bedrock, Databricks, NVIDIA NIM, Ollama, self-hosted transformers, …).
embedding_model	no	string	—	—	Embedding model id (e.g. ‘text-embedding-3-small’, ‘voyage-3.5’, ‘embed-english-v3.0’).
embedding_api_key	no	string	—	—	API key for the embedding provider. Stored encrypted at rest.
embedding_dimensions	no	integer	min 1; max 8192	—	Optional sanity-check on returned vector size. When set, vectors of wrong length are rejected before sending to the store. Connection-test fills this in automatically when it can probe the store.
azure_openai_endpoint	no	string	—	—	Azure OpenAI resource endpoint, e.g. ‘https://my-resource.openai.azure.com’. Required when embedding_provider=‘azure-openai’.
azure_openai_deployment	no	string	—	—	Azure deployment name (NOT the model — deployments map to models inside Azure). Required when embedding_provider=‘azure-openai’.
azure_openai_api_version	no	string	—	—	Azure OpenAI API version, e.g. ‘2024-02-15-preview’. Defaults to ‘2024-02-15-preview’ when blank.
custom_endpoint	no	string	—	—	Full URL of the embeddings endpoint. http(s) only; private / loopback / cloud-metadata hosts are rejected as an SSRF guard. Required when embedding_provider=‘custom’.
custom_auth_header_name	no	string	—	—	Auth header name. Defaults to ‘Authorization’ when blank.
custom_auth_header_value	no	string	—	—	Auth header value template. Use ‘{{api_key}}’ as a placeholder for the embedding_api_key. Example: ‘Bearer {{api_key}}’. Defaults to ‘Bearer {{api_key}}’.
custom_request_template	no	string	—	—	JSON request body template. Substitutions: ‘{{text}}’ (query string) and ‘{{model}}’ (embedding_model). Default ’{“input”: “{{text}}”, “model”: “{{model}}”}’.
custom_response_path	no	string	—	—	Dotted path into the JSON response to find the vector. Use ’[]’ to mean ‘first array element’. Example: ‘data.[].embedding’ (OpenAI shape), ‘embeddings.[]’ (Cohere v1). Default ‘data.[].embedding’.
bedrock_region	no	string	—	—	AWS region of the Bedrock Knowledge Base, e.g. ‘us-east-1’.
bedrock_knowledge_base_id	no	string	—	—	Bedrock Knowledge Base id (e.g. ‘ABCD1234’).
bedrock_search_type	no	enum	one of HYBRID, SEMANTIC	HYBRID	Retrieval mode for Bedrock: HYBRID (keyword + vector) or SEMANTIC (vector only).
bedrock_model_arn	no	string	—	—	Foundation-model ARN used to generate the grounded answer. Required for kind=‘bedrock_retrieve_and_generate’.
bedrock_aws_access_key_id	no	string	—	—	IAM access key id with bedrock:Retrieve / bedrock:RetrieveAndGenerate on the KB. An identifier (stored as-is).
bedrock_aws_secret_access_key	no	string	—	—	IAM secret access key. Stored encrypted at rest; UI reads return ‘REDACTED’.
bedrock_aws_session_token	no	string	—	—	Optional STS session token for temporary credentials. Stored encrypted at rest.
azure_search_endpoint	no	string	—	—	Azure AI Search service endpoint, e.g. ‘https://my-svc.search.windows.net’.
azure_search_index	no	string	—	—	Azure AI Search index name to query.
azure_search_api_version	no	string	—	—	REST api-version. Defaults to a recent stable version when blank.
azure_search_query_type	no	enum	one of simple, semantic	simple	’simple’ = standard keyword ranking; ‘semantic’ = Azure’s cross-encoder L2 reranker (needs a semantic configuration).
azure_search_semantic_config	no	string	—	—	Semantic configuration name on the index. Required when azure_search_query_type=‘semantic’.
azure_search_api_key	no	string	—	—	Azure AI Search admin or query API key. Stored encrypted at rest.
vertex_project_id	no	string	—	—	GCP project id hosting the Vertex AI Search app/data store.
vertex_location	no	string	—	global	Vertex AI Search location: ‘global’ (default), ‘us’, or ‘eu’.
vertex_engine_id	no	string	—	—	Vertex AI Search engine (app) id or data store id.
vertex_serving_config	no	string	—	default_search	Serving config id. Defaults to ‘default_search’.
vertex_mode	no	enum	one of search, answer	search	’search’ = ranked documents/snippets; ‘answer’ = a grounded answer with citations.
vertex_service_account_json	no	string	—	—	Service-account JSON (whole file contents) with Discovery Engine access. Stored encrypted at rest; used to mint a short-lived OAuth2 access token.
openai_vector_store_id	no	string	—	—	OpenAI Vector Store id, e.g. ‘vs_abc123’.
openai_base_url	no	string	—	—	Optional base URL override (defaults to https://api.openai.com). http(s) only; private/loopback/metadata hosts blocked.
openai_rewrite_query	no	boolean	—	false	When true, OpenAI rewrites the natural-language query before searching.
openai_api_key	no	string	—	—	OpenAI API key with vector-store read access. Stored encrypted at rest.
obsidian_repo	no	string	—	—	GitHub repository holding the Obsidian vault, as “owner/name” (synced via the Obsidian Git plugin).
obsidian_branch	no	string	—	—	Branch to read. Defaults to ‘main’ when blank.
obsidian_vault_subpath	no	string	—	—	Optional folder inside the repo that holds the vault (blank = repo root).
obsidian_token	no	string	—	—	GitHub fine-grained PAT with Contents read access (write needed for note commits). Stored encrypted at rest; UI reads return ‘REDACTED’.
obsidian_enable_tools	no	boolean	—	false	Also register the structured obsidian_* tools (search / get_note / list_bases / run_base) so the agent can navigate frontmatter, tags, backlinks, and Bases views of this vault.
obsidian_allow_write	no	boolean	—	false	Additionally register obsidian_write_note so the agent can commit notes to the vault. Requires a token with Contents write access.

service_sources items

Configured SaaS service data sources (Services card). Each pulls documents/records through a connector added on the /connectors page and ingests them into this scope’s retrieval index. Sync-state fields (last_synced_at, last_sync_status, last_sync_error, item_count) are server-stamped and never trusted from clients.

Field	Required	Type	Allowed values / constraints	Default	Description
id	yes	string	pattern ^svc_[a-z0-9]+$	—	Stable source id; ingested files reference it via service_source_id.
name	yes	string	max len 120	—	Display name shown on the data page.
connector_kind	yes	enum	one of composio, mcp, openapi, graphql, s3, native	—	Which connector mechanism this source pulls through.
connector_ref	yes	string	max len 200	—	Composio record id (stringified number) or connector name from the connectors section.
service	yes	string	max len 64	—	Recipe key in config/service-ingest-catalog.json (e.g. ‘googledrive’, ‘jira’), or ‘custom’.
object_type	no	string	max len 64	—	Object-type key within the recipe (e.g. ‘files’, ‘issues’).
query	no	string	max len 2000	—	Recipe-specific listing filter (Drive query, JQL, SOQL WHERE, search text…).
extra_args_json	no	string	max len 4000	—	JSON object string merged into the list tool’s arguments — for recipes whose tools need per-source identifiers (Sentry org slug, Bitbucket workspace/repo, BambooHR subdomain…). GraphQL: becomes the query variables.
include_terms	no	string	max len 1000	—	Comma-separated terms — when set, an item’s title must contain one to be ingested.
exclude_terms	no	string	max len 1000	—	Comma-separated terms — items whose title contains one are skipped.
max_items	no	integer	min 1	—	Per-source item cap; clamped to the plan’s service_import_item_limits.
sync	yes	enum	one of manual, daily, weekly, monthly	—	Automatic re-sync cadence; next sync is due last_synced_at + period.
chunking_config	no	object	—	—	Same shape as files[].chunking_config; inherited by every file this source ingests.
custom	no	object	—	—	Owner-authored field mapping when service === ‘custom’ (build-your-own connector). Item fields below.
last_synced_at	no	string	—	—	Server-stamped ISO time of the last completed sync.
last_sync_status	no	enum	one of ok, error, “	—
last_sync_error	no	string	max len 2000	—
item_count	no	integer	—	—	Files currently held by this source.
s3	no	object	—	—	Direct S3 / S3-compatible bucket config (connector_kind ‘s3’). secret_access_key is encrypted at rest and read back as ‘REDACTED’. Item fields below.

files.chunking_config fields

Configuration object controlling how the file is split into chunks for RAG retrieval. Passed to the ingest pipeline; controls chunk size, overlap, strategy, metadata stamping, and contextual enrichment.

Field	Required	Type	Allowed values / constraints	Default	Description
chunk_size	no	integer	—	—	Target chunk size in tokens (recommended 256–1024). Chunks smaller than this may result from sentence/paragraph boundaries depending on split_strategy.
chunk_overlap_pct	no	integer	—	—	Overlap between consecutive chunks as a percentage of chunk_size (0–50). Default 0; higher values improve recall for information spanning chunk boundaries at the cost of duplicate embeddings.
split_strategy	no	enum	one of auto, fixed_token, recursive, semantic, structural	—	Splitting algorithm: ‘auto’ (hybrid structural splitting with recursive fallback), ‘fixed_token’ (strict token slicing), ‘recursive’ (paragraph→sentence→token), ‘semantic’ (embedding-distance shifts), or ‘structural’ (markdown headings / CSV rows). ‘auto’ is recommended.
metadata	no	object	—	—	Object defining which source-attribution fields are stamped on each chunk. Includes toggles for source path, section title, page number, author, and document date — all optional and default false except source/section_title which default true.
contextual_enrichment	no	object	—	—	Object controlling whether auto-generated context headers are prepended to each chunk before embedding (a contextual-retrieval technique). Opt-in; enabled=true incurs one LLM call per chunk. Includes toggles for doc title, section path, and optional custom template.
preview_before_ingest	no	boolean	—	—	When true, the file is converted and chunked but PAUSES before indexing so the owner can preview and optionally edit chunks on the preview page, then clicks ‘Ingest these chunks’ to finish. Default false.

files.doc_profile fields

LLM-generated knowledge profile written by the ingest pipeline (not owner-editable): what the document is about + topic keywords. Feeds the agent’s document catalog so it can route to the right document.

Field	Required	Type	Allowed values / constraints	Default	Description
summary	no	string	max len 500	—	2–3 sentence summary of what the document is about and what kinds of questions it answers.
keywords	no	array of string	max 8 items	—	Up to 8 topic keywords / entity names, most distinctive first.
profiled_at	no	string	—	—	ISO timestamp of the profiling run.

service_sources.custom fields

Owner-authored field mapping when service === ‘custom’ (build-your-own connector).

Field	Required	Type	Allowed values / constraints	Default	Description
list_tool	yes	string	max len 4000	—	Connector tool (Composio/MCP name, OpenAPI operationId) that lists items — or, for GraphQL connectors, the full query document.
list_args_json	no	string	max len 4000	—	JSON object string of extra arguments for the list tool.
items_path	no	string	max len 200	—	Dot-path to the items array in the result; empty = auto-discover.
id_path	no	string	max len 200	—
title_path	no	string	max len 200	—
body_path	no	string	max len 200	—
url_path	no	string	max len 200	—
updated_path	no	string	max len 200	—

service_sources.s3 fields

Direct S3 / S3-compatible bucket config (connector_kind ‘s3’). secret_access_key is encrypted at rest and read back as ‘REDACTED’.

Field	Required	Type	Allowed values / constraints	Default	Description
bucket	yes	string	max len 255	—
region	no	string	max len 64	—
endpoint	no	string	max len 400	—	Custom https endpoint for R2 / Spaces / MinIO; empty = AWS.
prefix	no	string	max len 512	—	Only objects under this key prefix are listed.
access_key_id	no	string	max len 256	—
secret_access_key	no	string	max len 2048	—