Platform Guide

Datasets & enterprise connectors

Datasets are tenant-scoped containers over uploads and live connectors, with scheduled sync, change reconciliation, and permission snapshots that flow forward to every query.

What a dataset is

A dataset is the unit of ingestion and permissions. It has an id, a name, a connector type, and a set of sources (e.g. a SharePoint site, a Drive folder, an S3 prefix, a mailbox). Tenancy is enforced at the dataset layer — a dataset belongs to one tenant and only that tenant's principals can query it. Routes live in src/lace/api/routes/datasets.py plus per-connector files (dataset_sharepoint_connectors.py, dataset_google_drive_connectors.py, etc.).

terminalbash

POST /v1/datasets
{ "name": "Q3 Contracts", "connector_type": "sharepoint" }
# → { dataset_id, status:"ready" }

POST /v1/datasets/{id}/sharepoint/connect
{ "site_url": "https://acme.sharepoint.com/sites/legal", "resource": "/drive/root" }

POST /v1/datasets/{id}/sync  # ad-hoc, or wait for scheduled sync
GET  /v1/datasets/{id}/items?limit=20

Supported connectors

Connector	Type	Incremental? / deletes? / ACLs?
SharePoint	`sharepoint`	yes / yes / yes — Graph delta
Google Drive	`google_drive`	yes / yes / yes — changes API + permissions list
OneDrive	`onedrive`	yes / yes / yes
Dropbox	`dropbox`	yes / yes / yes
Notion	`notion`	yes / yes / no (page-level)
Gmail / Outlook	`gmail` / `outlook_mail`	yes / yes / mailbox ACL
Slack	`slack` (via connector marketplace)	yes / yes / channel ACL
S3	`s3`	yes (ETag) / yes / bucket policy → snapshot
Web / scraper	`web`	yes (ETag/Last-Modified) / no / n/a
Databases	SQL CDC (postgres/mysql)	CDC log / yes / row-level

Each connector reports its own capability manifest (src/lace/connectors/datasets/registry.py). It tells the platform whether the connector supports incremental sync, delete propagation, and permission reads, so the platform never assumes a lowest common denominator. Capability gaps degrade gracefully instead of silently dropping deletes or leaking restricted docs.

Sync, scheduling & reconciliation

Sync is driven by src/lace/connectors/datasets/sync_engine.py + scheduled_sync.py:

Scheduled sync — cron per source (POST /v1/datasets/{id}/scheduled-sync); managed by connector_sync.py and scheduler.py.
Ad-hoc sync — POST /v1/datasets/{id}/sync for a one-off crawl.
Leases & retries — source leases (source_operation_lease.py) prevent double-crawl; retry policy (retry_policy.py) backs off on throttling.
Reconciliation — sync diffs against what was previously indexed: edits update in place, deletes propagate to the index and the KG, moves keep stable document / block identity. That stability is why citations stay durable.

Permission snapshots

A connector does not only bring content. It also brings the source system's access rules as an ACL snapshot per item. Snapshots are versioned alongside the content and flow forward to every query. Retrieval and enterprise search filter by them at query time (SQL prefilter plus Python finalize) so facet counts never leak a restricted entity. See src/lace/domain/enterprise_search/acl.py.

Ingest — parse, chunk, embed, index

After sync, ingest parses (with docx/xlsx/pdf handling), chunks with stable block identity, embeds, and indexes into Postgres + pgvector (src/lace/domain/ingest, src/lace/domain/parsing). Document / block / span ids are stable across re-ingests — an evidence pointer taken last quarter still resolves.

Operating datasets

Browse sources: GET /v1/datasets/{id}/sources and POST /v1/datasets/{id}/sources/browse.
Admin UI: dataset admin at /admin/datasets (see src/lace/api/routes/dataset_connector_admin.py).
Connector marketplace: src/lace/connectors/marketplace.py lists what is enabled for your footprint (cloud / VPC / air-gapped differs).

Next: search & RAG or Enterprise Search product.