> ## Documentation Index
> Fetch the complete documentation index at: https://infino.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Connect & storage: memory, disk, S3, and Azure

> Run Infino against in-memory, local disk, or object storage backends like S3 and Azure Blob, picked at runtime by the connect URI scheme.

Where your data lives is chosen by the `connect` URI:

```python Python icon="python" theme={null}
db = infino.connect("memory://")              # in-process, ephemeral
db = infino.connect("./data")                 # local disk (durable)
db = infino.connect("s3://bucket/prefix")     # object storage (durable)
```

```typescript Node.js icon="node-js" theme={null}
const db = connect("memory://");              // in-process, ephemeral
const db = connect("./data");                 // local disk (durable)
const db = connect("s3://bucket/prefix");     // object storage (durable)
```

```rust Rust icon="rust" theme={null}
use infino::connect;

let db = connect("memory://")?;               // in-process, ephemeral
let db = connect("./data")?;                  // local disk (durable)
let db = connect("s3://bucket/prefix")?;      // object storage (durable)
```

Data is stored as standard Apache Parquet at that location, so anything that reads Parquet
can read it.

> [!WARNING]
>
> `memory://` is in-process and ephemeral. `update` and `delete` require **durable**
> storage (a path or `s3://` URI).

## Object-storage configuration

Credentials go in `storage_options` (`storageOptions` in Node.js), keyed by the
standard [`object_store`](https://docs.rs/object_store) config strings — `aws_*`
for S3, `azure_*` for Azure Blob (the same names the AWS and Azure SDKs use).
Infino reads no credentials from the environment; omit them to use ambient cloud
identity (an IAM instance role or Azure managed identity).

```python Python icon="python" theme={null}
# S3
db = infino.connect(
    "s3://bucket/prefix",
    storage_options={
        "aws_access_key_id": "...",
        "aws_secret_access_key": "...",
        "aws_region": "us-east-1",
    },
    cache_dir="/var/cache/infino",   # local disk cache for remote-backed tables
    cache_budget_bytes=2_000_000_000,
    validate=True,                   # probe the store now, not on first query
)

# Azure Blob
db = infino.connect(
    "az://container/prefix",
    storage_options={
        "azure_storage_account_name": "...",
        "azure_storage_account_key": "...",
    },
)
```

```typescript Node.js icon="node-js" theme={null}
// S3
const db = connect("s3://bucket/prefix", {
  storageOptions: {
    aws_access_key_id: "...",
    aws_secret_access_key: "...",
    aws_region: "us-east-1",
  },
  cacheDir: "/var/cache/infino",   // local disk cache for remote-backed tables
  cacheBudgetBytes: 2_000_000_000,
  validate: true,                  // probe the store now, not on first query
});

// Azure Blob
const db = connect("az://container/prefix", {
  storageOptions: {
    azure_storage_account_name: "...",
    azure_storage_account_key: "...",
  },
});
```

```rust Rust icon="rust" theme={null}
use infino::{connect_with, ConnectOptions};

// S3
let db = connect_with(
    "s3://bucket/prefix",
    ConnectOptions::new()
        .with_storage_option("aws_access_key_id", "...")
        .with_storage_option("aws_secret_access_key", "...")
        .with_storage_option("aws_region", "us-east-1")
        .with_cache_dir("/var/cache/infino")     // local disk cache for remote-backed tables
        .with_cache_budget_bytes(2_000_000_000)
        .with_validate(true),                    // probe the store now, not on first query
)?;

// Azure Blob
let db = connect_with(
    "az://container/prefix",
    ConnectOptions::new()
        .with_storage_option("azure_storage_account_name", "...")
        .with_storage_option("azure_storage_account_key", "..."),
)?;
```

Common credential keys:

| Backend | Keys |
| - | - |
| S3 | `aws_access_key_id`, `aws_secret_access_key`, `aws_region`, `aws_session_token`, `aws_endpoint` |
| Azure | `azure_storage_account_name`, `azure_storage_account_key`, `azure_storage_sas_key`, `azure_storage_client_id`, `azure_storage_client_secret`, `azure_storage_tenant_id` |

The full set is whatever `object_store` accepts for the backend; an unknown key
is rejected at `connect`. For an S3-compatible endpoint (MinIO / R2 / Ceph), set
`aws_endpoint` (add `aws_allow_http: "true"` for plain HTTP) alongside the
credentials.

> [!NOTE]
>
> Pass `validate` (`with_validate(true)` in Rust) to probe the backend during
> `connect`, so wrong credentials or an unreachable bucket fail there instead of
> on the first query. The local cache keeps hot superfiles on disk so warm queries
> don't re-fetch from object storage.

## Connect options

| Option (Python / Node.js / Rust) | Description |
| - | - |
| `storage_options` / `storageOptions` / `with_storage_option(k, v)` | cloud credentials & tuning, keyed by `object_store`'s `aws_*` / `azure_*` / `google_*` config strings |
| `validate` / `validate` / `with_validate(bool)` | probe the object store at `connect` to fail fast on bad credentials (default off) |
| `cache_dir` / `cacheDir` / `with_cache_dir` | local disk-cache directory for remote-backed tables |
| `cache_budget_bytes` / `cacheBudgetBytes` / `with_cache_budget_bytes` | disk-cache budget, in bytes |
| `connection_memory_budget_bytes` / `connectionMemoryBudgetBytes` / `with_connection_memory_budget_bytes` | per-connection heap budget, in bytes (`0` / omitted = measure only) |
| `cold_fetch_mode` / `coldFetchMode` / `with_cold_fetch_mode` | how cold-cache misses are serviced |
| `api_key` / `apiKey` / `with_api_key` | API key for a hosted `https://<host>/<database>` connection; falls back to `INFINO_API_KEY` |

## Connection memory budget

`connection_memory_budget_bytes` caps the **RAM (heap)** a connection may hold while it
works, the transient memory used to ingest data and run queries over it (keyword, vector,
hybrid, or SQL). It is **per connection**: one ceiling shared by every table on the
connection and by all its work, tracked as a single running total. By default it is unset,
usage is measured but never enforced; pass a byte value to enforce it.

When a request would cross the limit, Infino tries to stay within budget before it fails:

* **SQL that can spill** (sorts, aggregations, joins) writes intermediate state to disk and
  keeps going, so the query still completes, just slower.
* **Work that can't spill** (a vector search's cold fetch, an ingest build) is refused
  before it allocates, returning a recoverable error rather than risking an out-of-memory
  crash.

```python Python icon="python" theme={null}
db = infino.connect("./data", connection_memory_budget_bytes=512_000_000)
```

```typescript Node.js icon="node-js" theme={null}
const db = connect("./data", { connectionMemoryBudgetBytes: 512_000_000 });
```

```rust Rust icon="rust" theme={null}
let db = connect_with(
    "./data",
    ConnectOptions::new().with_connection_memory_budget_bytes(512_000_000),
)?;
```

A refusal surfaces as each language's idiomatic recoverable error: `ConnectionMemoryBudgetError`
in Python, a thrown `Error` in Node, and `InfinoError::OverBudget` in Rust. Catch it, then
narrow the query, split the ingest, or raise the budget.

> [!NOTE]
>
> This budget is RAM only. It is separate from `cache_budget_bytes`, which bounds the local
> disk cache, and it does not cap memory-mapped reads (the OS reclaims those on its own).
> Give each independent workload its own connection so their budgets stay isolated rather
> than sharing one ceiling. Omit it (or pass `0`) to only measure usage, never enforce.

## Limitations

* **`memory://` is ephemeral** and can't be mutated. `update`/`delete` need a path or `s3://` URI.
* **Object-storage reads are cached locally.** Cold queries fetch from the store; size the cache (`cache_budget_bytes`) for your working set.

## See also

* [Tables](https://infino.ai/docs/guides/tables.md)
* [Open format](https://infino.ai/docs/guides/parquet-interop.md)
* [FAQ](https://infino.ai/docs/faq.md)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.