Buckets

Key Concepts

Buckets is built around a few core primitives that work together to give your organization a structured, searchable file repository.

Buckets & Folders

A bucket is the top-level container — it belongs to an organization and is the unit you share with other services. Inside a bucket, folders organize files and can be nested into arbitrary directory trees. Buckets and folders carry an optional pipeline_config that defines sparse canonical-v2 preprocessing, OCR, analysis, grouping, evidence, and embedding settings. Folder values deep-merge over bucket values; omitted nested fields inherit while explicit false, null, and empty lists override. Single bucket and folder reads preserve that stored sparse value as pipeline_config and also return the fully effective resolved_pipeline_config.

Files

A file represents a single uploaded file — a PDF, image, text file, or any other supported format. Each file tracks its original filename, content_type, and a status field that reflects where it is in the processing lifecycle. Files are always assigned to a folder and can carry a free-text description for additional context.

File Associations

Associations link a file to external entities in your system. Each association is a pair of entity_type and entity_id — for example, linking an invoice PDF to a workflow item or a support ticket. You can attach associations at upload time or add and remove them later via the update endpoint. This makes it easy to find every file related to a given entity, regardless of which folder it lives in.

Processing Runs

A processing run is a batch operation that takes one or more files through the ingestion pipeline. Each run has its own run_id, tracks its status (pending, processing, completed, or failed), and stores the effective configuration snapshot used. New snapshots are canonical v2; historical v1 snapshots remain readable. Results retain adaptive route provenance, structured schema values, optional evidence locations, semantic grouping confidence/fallback metadata, and embeddings.

Ingestion Pipeline

The ingestion pipeline transforms raw files into structured, searchable data. The flow starts with uploading a file into a folder, then optionally triggering a processing run that moves the file through several stages.

Upload Flow

To upload a file, first request a presigned upload URL from the API. This returns a time-limited URL that your client uses to upload the file directly to storage — keeping large files off the API server. Once the upload completes, confirm it to create the file record.

# Step 1 — Request a presigned upload URL
curl -X POST https://platform.ergondata.ai/api/v1/buckets/buckets/{bucket_id}/upload-url \
  -H "Authorization: Bearer {token}" \
  -H "Content-Type: application/json" \
  -d '{
    "filename": "contract.pdf",
    "content_type": "application/pdf",
    "size": 204800
  }'

# Step 2 — Upload the file directly using the returned URL
curl -X PUT "{upload_url}" \
  -H "Content-Type: application/pdf" \
  --data-binary @contract.pdf

# Step 3 — Confirm the upload to create the file record
curl -X POST https://platform.ergondata.ai/api/v1/buckets/buckets/{bucket_id}/confirm \
  -H "Authorization: Bearer {token}" \
  -H "Content-Type: application/json" \
  -d '{
    "object_key": "{object_key}",
    "filename": "contract.pdf",
    "content_type": "application/pdf",
    "size": 204800
  }'

Processing Stages

Once a file exists, processing selects work by page and by configured gates:

• Preprocess and OCR routing — text_source records native, OCR, or no text independently from the text, vision, or hybrid analysis_input. OCR mode can be auto, always, or off; forms, tables, signatures, and evidence locations can add OCR/raster cost.
• Analysis — Descriptions and typed schemas are independent gates. Schema values can include source evidence only when evidence locations are enabled.
• Semantic page grouping — A text-first boundary pass records confidence and reasons, with bounded visual fallback for uncertain boundaries.
• Embedding generation — Runs independently from analysis and can use extracted text, descriptions, or both.

Folder-Level Configuration

Set a sparse canonical-v2 pipeline_config on a bucket or folder. Resolution deep-merges the bucket and folder ancestry before ordinary cost-lean defaults are applied: OCR auto, descriptions/schemas/grouping off, and embeddings off. Conversation-managed uploads intentionally enable auto-processing and extracted-text embeddings.

Triggering Processing

To process files manually, call the process endpoint with file IDs and an optional deep sparse v2 override. Omitted nested values inherit; false, null, and empty lists are authoritative for this run only.

curl -X POST https://platform.ergondata.ai/api/v1/buckets/files/process \
  -H "Authorization: Bearer {token}" \
  -H "Content-Type: application/json" \
  -d '{
    "file_ids": ["{file_id}"],
    "version": 2,
    "ocr": { "mode": "auto", "tables": true },
    "analysis": {
      "grouping": "semantic",
      "evidence_locations": true
    },
    "embedding": { "enabled": true, "source": "extracted_text" }
  }'

The response returns a run_id you can use to poll for status. Once the run completes, fetch results for each file to see classifications, extracted fields, and metadata.

Inspecting Runs

Open a file in the bucket browser and select Processing to inspect any recent run. The live linear pipeline shows pending, running, completed, skipped, and failed stages. Expand a stage to inspect its deduplicated actions, start and end times, durations, delivery and operation attempts, routing and cache counts, and sanitized failure categories. Historical runs created before structured tracing show an empty-trace state. The run selector in the dialog header scopes the entire inspector: results, schemas, semantic groups, highlights, and processing telemetry all change together while the original uploaded file remains stable.

curl https://platform.ergondata.ai/api/v1/buckets/runs/{run_id} \
  -H "Authorization: Bearer {token}"

Semantic Search

After files have been processed with embeddings enabled, their text is chunked into passages and converted into vector representations. This powers natural-language search across your entire file library — you describe what you're looking for in plain language, and the API returns the most relevant passages ranked by similarity.

How Files Are Embedded

During processing, extracted text is split into overlapping chunks sized for optimal retrieval. Each chunk is passed through an embedding model to produce a dense vector. These vectors are stored alongside the chunk text and file metadata, enabling fast similarity lookups at query time.

Querying

To search, send a natural-language query to the search endpoint. The query itself is embedded using the same model, then compared against all stored chunks. Results are ranked by cosine similarity and filtered by a configurable threshold (default 0.3) to exclude weak matches. Each result includes the matching text snippet, the file and folder it belongs to, and the similarity score.

curl "https://platform.ergondata.ai/api/v1/buckets/search/files?q=payment%20terms&limit=5&threshold=0.35" \
  -H "Authorization: Bearer {token}"

Search results are scoped to files the authenticated caller has access to. You can control the number of results with limit and adjust the minimum similarity with threshold to balance precision and recall.

Cross-Service Grants

A bucket can be shared with other principals — members, agents, or another service's resource — so they can read, upload, or search files on behalf of their own features. Sharing uses the platform's unified access model: every bucket and folder is a scoped resource, and access is granted to a principal with a specific permission.

Granting Access

To share a bucket, create an access grant on it that names the principal and the permission it should hold. List the principals that can be granted with GET .../access/eligible and the available permissions with GET .../access/resource-types.

curl -X POST https://platform.ergondata.ai/api/v1/buckets/buckets/{bucket_id}/access/grants \
  -H "Authorization: Bearer {token}" \
  -H "Content-Type: application/json" \
  -d '{
    "principal_type": "agent",
    "principal_id": "{principal_id}",
    "permission_id": "{permission_id}",
    "effect": "allow"
  }'

Connection Requests

Access can also be initiated from the other side: a principal that wants to use a bucket can raise a connection request, which the bucket owner then approves or rejects. This keeps the owner in control while letting consumers ask for what they need.

# List pending requests for a bucket
curl "https://platform.ergondata.ai/api/v1/buckets/buckets/{bucket_id}/access/connection-requests?status=pending" \
  -H "Authorization: Bearer {token}"

# Approve one
curl -X POST https://platform.ergondata.ai/api/v1/buckets/buckets/{bucket_id}/access/connection-requests/{request_id}/approve \
  -H "Authorization: Bearer {token}"

# List the buckets that are connected (inbound) to this one
curl "https://platform.ergondata.ai/api/v1/buckets/buckets/{bucket_id}/access/connections" \
  -H "Authorization: Bearer {token}"

Common scenarios include sharing buckets with Workflows for file attachment storage, Agent Hub for giving agents access to a knowledge base, and any integration that needs scoped file access.