![\"DataFusion](\"https://datafusion.apache.org/blog/images/datafusion-51.0.0/performance_over_time_clickbench.png\")
Announcing the release of Spice v2.0-rc.2! π₯
\nv2.0.0-rc.2 is the second release candidate for advanced testing of v2.0, building on v2.0.0-rc.1.
\nHighlights in this release candidate include:
\narrow-rs, datafusion-federation, and datafusion-table-providersPARTITION BY Support for Cayenne enabling SQL-defined partitioning in CREATE TABLE statementssoda (Socrata Open Data) format support, json_pointer extraction, and auto-detectionDistributed query for Cayenne-backed tables now has better partition awareness for both reads and writes.
\nKey improvements:
\nDoPut ingestion now splits partitioned Cayenne writes and forwards them to the responsible executors instead of routing through a single raw-forward path.runtime.task_history: Task history is now replicated across the distributed cluster for observability.Spice now supports MERGE INTO statements for Cayenne catalog tables, enabling upsert-style data operations with full distributed support.
Key improvements:
\nMERGE INTO statements against Cayenne catalog tables for combined insert/update/delete operations.PARTITION BY Support for CayenneβSQL Partition Management: Spice now supports PARTITION BY for Cayenne-backed CREATE TABLE statements, enabling partition definitions to be expressed directly in SQL and persisted in the Cayenne catalog.
Key improvements:
\nCREATE TABLE ... PARTITION BY (...).CREATE TABLE is distributed to executors in cluster mode.Utf8View.Example:
\nCREATE TABLE events (id INT, region TEXT, ts TIMESTAMP) PARTITION BY (region)
Spice now includes additional catalog connectors for major database systems, improving schema discovery and federation workflows across external data systems.
\nKey improvements:
\ninformation_schema / INFORMATION_SCHEMA to discover schemas and tables.Example PostgreSQL catalog configuration:
\ncatalogs:
- from: pg
name: pg
include:
- 'public.*'
params:
pg_host: localhost
pg_port: 5432
pg_user: postgres
pg_pass: ${secrets:POSTGRES_PASSWORD}
pg_db: my_database
pg_sslmode: disable
JSON ingestion is now more flexible and robust.
\nKey improvements:
\njson_pointer Extraction: Extract nested payloads before schema inference and reading using RFC 6901 JSON Pointer syntax.Example using json_pointer to extract nested data from an API response:
datasets:
- from: https://api.example.com/v1/data
name: users
params:
json_pointer: /data/users
Apache DataFusion has been upgraded from v52.2.0 to v52.4.0, with aligned updates across arrow-rs, datafusion-federation, and datafusion-table-providers.
Key improvements:
\ntry_cast_to now uses strict cast to return errors on overflow instead of silently producing NULL values.FlightSQLExec.| Dependency | Version / Update |
|---|---|
| Turso (libsql) | v0.5.3 (from v0.4.4) |
| iceberg-rust | v0.9 |
| Vortex | Map type support, stack-safe IN-lists |
| arrow-rs | Arrow v57.2.0 |
| datafusion-federation | Updated for DataFusion v52.4.0 alignment |
| datafusion-table-providers | Updated for DataFusion v52.4.0 alignment |
| datafusion-ballista | Bumped to fix BatchCoalescer schema mismatch panic |
Cayenne released as RC: Cayenne data accelerator is now promoted to release candidate status.
\nFile Update Acceleration Mode: Added mode: file_update acceleration mode for file-based data refresh.
spice completions Command: New CLI command for generating shell completion scripts, with auto-detection of shell directory.
--endpoint Flag: Added --endpoint flag to spice run with scheme-based routing for custom endpoints.
mTLS Client Auth: Added mTLS client authentication support to the spice sql REPL.
DynamoDB DML: Implemented DML (INSERT, UPDATE, DELETE) support for the DynamoDB table provider.
\nCaching Retention: Added retention policies for cached query results.
\nGraphQL Custom Auth Headers: Added custom authorization header support for the GraphQL connector.
\nClickHouse Date32 Support: Added Date32 type support for the ClickHouse connector.
AWS IAM Role Source: Added iam_role_source parameter for fine-grained AWS credential configuration.
S3 Metadata Columns: Metadata columns renamed to _location, _last_modified, _size for consistency, with more robust handling in projected queries.
S3 URL Style: Added s3_url_style parameter for S3 connector URL addressing (path-style vs virtual-hosted). Useful for S3-compatible stores like MinIO:
params:
s3_endpoint: https://minio.local:9000
s3_url_style: path
S3 Parquet Performance: Improved S3 parquet read performance.
\nHTTP Caching: Transient HTTP error responses such as 429 and 5xx are no longer cached, preventing stale error payloads from being served from cache.
HTTP Connector Metadata: Added response_headers as structured map data for HTTP datasets.
Views on_zero_results: Accelerated views now support on_zero_results: use_source to fall back to the source when no results are found:
views:
- name: sales_summary
sql: |
SELECT region, SUM(amount) as total
FROM sales
GROUP BY region
acceleration:
enabled: true
on_zero_results: use_source
Flight DoPut Ingestion Metrics: Added rows_written and bytes_written metrics for Flight DoPut / ADBC ETL ingestion.
EXPLAIN ANALYZE Metrics: Added metrics for EXPLAIN ANALYZE in FlightSQLExec.
\nScheduler Executor Metrics: Added scheduler_active_executors_count metric for monitoring active executors.
Query Memory Limit: Updated default query memory limit from 70% to 90%, with GreedyMemoryPool for improved memory management.
MetastoreTransaction Support: Added transaction support to prevent concurrent metastore transaction conflicts.
\nIceberg REST Catalog: Coerce unsupported Arrow types to Iceberg v2 equivalents in the REST catalog API.
\nCDC Cache Invalidation: Improved cache invalidation for CDC-backed datasets.
\nSpice.ai Connector Alignment: Parameter names aligned across catalog and data connectors for Spice.ai Cloud.
\nCayenne File Size: Cayenne now correctly respects the configured target file size (defaults to 128MB).
\nCayenne Primary Keys: Properly set primary_keys/on_conflict for Cayenne tables.
Turso Metastore Performance: Cached metastore connections and prepared statements for improved Turso and SQLite metastore performance.
\nTurso SQL Robustness: More robust SQL unparsing and date comparison handling for Turso.
\nDictionary Type Normalization: Normalize Arrow Dictionary types for DuckDB and SQLite acceleration.
\nGitHub Connector Resilience: Improved GraphQL client resilience, performance, and ref filter handling.
\nODBC Fix: Fixed ODBC queries silently returning 0 rows on query failure.
\nAnthropic Fixes: Fixed compatibility issues with Anthropic model provider.
\nv1/responses API Fix: The /v1/responses API now correctly preserves client instructions when system_prompt is set.
Shared Acceleration Snapshots: Show an error when snapshots are enabled on a shared acceleration file.
\nDistributed Mode Error Handling: Improved error handling for distributed mode and state_location configuration.
Helm Chart: Added support for ServiceAccount annotations and AWS IRSA example.
\nPerplexity Removed: Removed Perplexity model provider support.
\nRust v1.93.1: Upgraded Rust toolchain to v1.93.1.
\nlocation, last_modified, size to _location, _last_modified, _size.v1/evals API removed: The /v1/evals endpoint has been removed.To upgrade to v2.0.0-rc.2, use one of the following methods:
\nCLI:
\nspice upgrade v2.0.0-rc.2
Homebrew:
\nbrew upgrade spiceai/spiceai/spice
Docker:
\nPull the spiceai/spiceai:2.0.0-rc.2 image:
docker pull spiceai/spiceai:2.0.0-rc.2
For available tags, see DockerHub.
\nHelm:
\nhelm repo update
helm upgrade spiceai spiceai/spiceai --version 2.0.0-rc.2
AWS Marketplace:
\nSpice is available in the AWS Marketplace.
\ninstall-postgres action by @krinart in #9629postgres-accel in CI builds for benchmarks by @sgrebnov in #9649response_headers by @krinart in #9670v1/evals functionality by @Jeadie in #9420on_zero_results: use_source for views by @krinart in #9699max_partitions_per_executor in allocate_initial_partitions by @Jeadie in #9659primary_keys/on_conflict for Cayenne tables by @krinart in #9739FlightSQLExec by @lukekim in #9882scheduler_active_executors_count and use it in spidapter by @Jeadie in #9885FallbackOnZeroResultsScanExec fallback plan by @sgrebnov in #9927BigQueryDialect more robust and add BigQuery TPC-H benchmark support by @lukekim in #9969spice completions auto-detects shell directory and writes file by @lukekim in #10068try_cast_to to error on overflow instead of silent NULL by @sgrebnov in #10104spice upgrade by @phillipleblanc in #10194runtime.params.parquet_page_index to Delta Lake connector by @sgrebnov in #10209Full Changelog: https://github.com/spiceai/spiceai/compare/v2.0.0-rc.1...v2.0.0-rc.2
", "url": "https://spiceai.org/releases/v2.0-rc.2", "title": "Spice v2.0-rc.2 (Apr 10, 2026)", "summary": "Announcing the release of Spice v2.0-rc.2! π₯", "date_modified": "2026-04-10T00:00:00.000Z", "author": { "name": "Evgenii Khramkov", "url": "https://github.com/ewgenius" }, "tags": [ "release", "distributed-query", "cayenne", "datafusion", "delta-lake", "databricks", "adbc", "data-connector" ] }, { "id": "https://spiceai.org/releases/v1.11.5", "content_html": "Announcing the release of Spice v1.11.5! π οΈ
\nSpice v1.11.5 is a patch release improving on_zero_results: use_source fallback performance, Delta Lake timestamp predicate data skipping, S3 Parquet read performance, PostgreSQL partitioned table support, Cayenne target file size handling, and preparing the CLI for v2.0 runtime upgrades.
on_zero_results: use_source Fallback Performance ImprovementβImproved the on_zero_results: use_source fallback path to run DataFusion's physical optimizer on the federated scan plan (#9927). The fallback path now runs SessionState::physical_optimizers() rules on the federated scan plan before execution, enabling parallel file group scanning and other optimizations. This results in significantly faster fallback queries on multi-core machines, particularly for file-based data sources like Delta Lake.
>= Timestamp PredicatesβDelta Lake table scans with >= timestamp filters now correctly prune files that do not match the predicate (#9932), improving query performance through more effective data skipping (file-level pruning).
The PostgreSQL data connector now supports partitioned tables (#9997) for both federated and accelerated queries.
\nImproved parquet read performance from S3 and other object stores (#10064), particularly for tables with many columns. Column data ranges are now coalesced into fewer, larger requests instead of being fetched individually, reducing the number of HTTP round-trips.
\nThe Cayenne accelerator now correctly respects the configured target file size (#10071). Previously, Cayenne could produce many small, fragmented Vortex files; with this fix, files are written at the expected target size, improving storage efficiency and query performance.
\nThe Spice CLI can now upgrade to v2.0 runtime versions. This enables upgrading to v2.0 release candidates and, once released, the v2.0 stable runtime.
\nspice upgrade v2.0.0-rc.1
Running spice upgrade without a version will upgrade to the latest stable version, including v2.0 once released.
\n\nNote: Native Windows runtime builds will no longer be provided in v2.0. Use WSL for local development instead.
\n
No breaking changes.
\nNo new cookbook recipes.
\nThe Spice Cookbook includes 86 recipes to help you get started with Spice quickly and easily.
\nTo upgrade to v1.11.5, use one of the following methods:
\nCLI:
\nspice upgrade
Homebrew:
\nbrew upgrade spiceai/spiceai/spice
Docker:
\nPull the spiceai/spiceai:1.11.5 image:
docker pull spiceai/spiceai:1.11.5
For available tags, see DockerHub.
\nHelm:
\nhelm repo update
helm upgrade spiceai spiceai/spiceai --version 1.11.5
AWS Marketplace:
\nSpice is available in the AWS Marketplace.
\nFallbackOnZeroResultsScanExec fallback plan by @sgrebnov in #9927>= timestamp predicates by @sgrebnov in #9932Full Changelog: https://github.com/spiceai/spiceai/compare/v1.11.4...v1.11.5
", "url": "https://spiceai.org/releases/v1.11.5", "title": "Spice v1.11.5 (Apr 1, 2026)", "summary": "Announcing the release of Spice v1.11.5! π οΈ", "date_modified": "2026-04-01T00:00:00.000Z", "author": { "name": "Sergei Grebnov", "url": "https://github.com/sgrebnov" }, "tags": [ "release", "s3", "cayenne", "delta-lake", "PostgreSQL" ] }, { "id": "https://spiceai.org/releases/v1.11.4", "content_html": "Announcing the release of Spice v1.11.4! β‘
\nSpice v1.11.4 is a patch release improving S3 metadata column query robustness and enabling on_zero_results: use_source for accelerated views.
on_zero_results: use_source SupportβAccelerated views now support the on_zero_results: use_source configuration (#9699). Previously, accelerated views only supported on_zero_results: return_empty, which returned an empty result set when the accelerated data contained no matching rows. With this change, views can fall back to querying the source data when the accelerated query returns zero results, matching the behavior already available for accelerated datasets.
Example configuration:
\nviews:
- name: sales_summary
sql: |
SELECT region, SUM(amount) as total
FROM sales
GROUP BY region
acceleration:
enabled: true
on_zero_results: use_source
When an accelerated view is configured with on_zero_results: use_source, the following happens at query time:
The accelerated store is queried first. The query runs against the view's accelerated data (e.g., Spice Cayenne, Arrow, DuckDB, or SQLite).
\nIf the accelerated query returns zero rows, the runtime falls back to re-executing the view's SQL query against the datasets it references.
\nReferenced datasets are queried according to their own configuration. The view's SQL is re-executed against each referenced dataset as it is configured. This means:
\non_zero_results: use_source and its accelerated store also returns zero rows, it will independently fall back to its own federated data source (e.g., Postgres, S3, etc.).This means the fallback can chain through multiple layers: first the view's acceleration, then each referenced dataset's acceleration, and finally the original data source β each layer independently applying its own on_zero_results behavior.
Example: Multi-layer fallback
\ndatasets:
- from: postgres:orders
name: orders
acceleration:
enabled: true
refresh_sql: \"SELECT * FROM orders WHERE created_at > now() - interval '7 days'\"
on_zero_results: use_source # Falls back to Postgres if accelerated data has no matches
views:
- name: recent_orders_summary
sql: |
SELECT status, COUNT(*) as order_count
FROM orders
GROUP BY status
acceleration:
enabled: true
on_zero_results: use_source # Falls back to re-running the SQL against referenced datasets
In this example, a query like SELECT * FROM recent_orders_summary WHERE status = 'cancelled' follows this path:
recent_orders_summary in the view's accelerated store (DuckDB/SQLite).SELECT status, COUNT(*) ... FROM orders GROUP BY status against the orders dataset.orders is accelerated, the query hits the orders accelerated store.orders also returns zero rows (e.g., the refresh_sql excluded cancelled orders), it falls back to querying Postgres directly.Improved the robustness of metadata column (location, last_modified, size) handling for S3 datasets. Building on the v1.11.3 release, this update addresses an additional edge case where the query optimizer's projection swap could cause an index out of bounds panic when metadata columns are referenced in projections with filters or scalar functions.
No breaking changes.
\nNo new cookbook recipes.
\nThe Spice Cookbook includes 86 recipes to help you get started with Spice quickly and easily.
\nTo upgrade to v1.11.4, use one of the following methods:
\nCLI:
\nspice upgrade
Homebrew:
\nbrew upgrade spiceai/spiceai/spice
Docker:
\nPull the spiceai/spiceai:1.11.4 image:
docker pull spiceai/spiceai:1.11.4
For available tags, see DockerHub.
\nHelm:
\nhelm repo update
helm upgrade spiceai spiceai/spiceai --version 1.11.4
AWS Marketplace:
\nSpice is available in the AWS Marketplace.
\non_zero_results: use_source for accelerated views by @krinart in #9699Full Changelog: https://github.com/spiceai/spiceai/compare/v1.11.3...v1.11.4
", "url": "https://spiceai.org/releases/v1.11.4", "title": "Spice v1.11.4 (Mar 12, 2026)", "summary": "Announcing the release of Spice v1.11.4! β‘", "date_modified": "2026-03-12T00:00:00.000Z", "author": { "name": "Sergei Grebnov", "url": "https://github.com/sgrebnov" }, "tags": [ "release", "s3", "data-connector", "acceleration" ] }, { "id": "https://spiceai.org/releases/v1.11.3", "content_html": "Announcing the release of Spice v1.11.3! π οΈ
\nSpice v1.11.3 is a patch release fixing schema consistency issues in the S3 and FlightSQL data connectors, improving CDC cache invalidation, and enhancing the HTTP data connector's error handling and response metadata.
\nFixed an issue where queries using metadata columns (location, last_modified, size) on S3 datasets produced Input field name does not match with the projection expression errors (#9647). This occurred when projecting metadata columns with filters or scalar functions (e.g., SELECT lower(location) FROM table WHERE location = '...'), and when projection returned no matching files.
Fixed an issue where the Flight SQL JDBC driver returned Unsupported ArrowType Utf8View errors when performing ::TEXT type casts (#9253). The FlightSQL endpoint now maps view types (e.g., Utf8View, BinaryView) to their non-view equivalents, ensuring compatibility with JDBC and ODBC clients.
Fixed an issue where the SQL results cache was invalidated on every change stream poll, even when zero records were returned (#9472). This caused near-total cache miss rates for datasets using refresh_mode: changes (e.g., DynamoDB Streams), effectively rendering the cache useless. Cache invalidation now only occurs when a change batch contains actual data changes.
response_headers column (Map type) to HTTP responses, providing access to response header metadata in query results.No breaking changes.
\nNo new cookbook recipes.
\nThe Spice Cookbook includes 86 recipes to help you get started with Spice quickly and easily.
\nTo upgrade to v1.11.3, use one of the following methods:
\nCLI:
\nspice upgrade
Homebrew:
\nbrew upgrade spiceai/spiceai/spice
Docker:
\nPull the spiceai/spiceai:1.11.3 image:
docker pull spiceai/spiceai:1.11.3
For available tags, see DockerHub.
\nHelm:
\nhelm repo update
helm upgrade spiceai spiceai/spiceai --version 1.11.3
AWS Marketplace:
\nSpice is available in the AWS Marketplace.
\nresponse_headers by @krinart in #9670Full Changelog: https://github.com/spiceai/spiceai/compare/v1.11.2...v1.11.3
", "url": "https://spiceai.org/releases/v1.11.3", "title": "Spice v1.11.3 (Mar 9, 2026)", "summary": "Announcing the release of Spice v1.11.3! π οΈ", "date_modified": "2026-03-09T00:00:00.000Z", "author": { "name": "Phillip LeBlanc", "url": "https://x.com/leblancphill" }, "tags": [ "release", "s3", "data-connector", "caching", "FlightSQL" ] }, { "id": "https://spiceai.org/releases/v2.0-rc.1", "content_html": "Announcing the release of Spice v2.0-rc.1! π
\nv2.0.0-rc.1 is the first release candidate for early testing of v2.0.
\nHighlights in this release candidate include:
\nCREATE TABLE and DROP TABLE via SQL for Iceberg and Cayenne catalogsSpice v2.0 includes several breaking changes. Review the breaking changes section before upgrading.
\nAI/ML support including local LLM/ML model and hosted LLM inference is now included in the default Spice build and image. The separate models build variant has been removed.
With models now included by default, the data-only distribution (without AI/ML support) is only published in nightly builds. Official production-ready data-only distributions are available exclusively through Spice Cloud and the Enterprise release.
\nA new Network Attached Storage (NAS) distribution with built-in SMB and NFS data connector support is also now available in nightly builds and with Spice.ai Enterprise.
\n| Distribution / Variant | Open Source | Spice Cloud | Enterprise |
|---|---|---|---|
| Default | β | β | β |
| Data | Nightly only | β | β |
| NAS (SMB + NFS) | Nightly only | β | β |
| Metal (macOS) | β | β | β |
| CUDA (Linux) | Nightly only | β | β |
| Allocator variants | Nightly only | β | β |
| ODBC connector | Local build only | β | β |
For more details, see the Distributions documentation.
\nDistributed Query exits Beta with active-active highly-available object-store-based distributed query.
\nDistributed query supports two execution modes:
\n/v1/queries API and results are materialized to object storage for later retrieval. Best for long-running analytical workloads, batch processing, and non-accelerated datasets in distributed mode.Key improvements:
\n/v1/sql and FlightSQL queries now execute synchronously on the scheduler when appropriate, reducing inter-node overhead for queries that don't benefit from distribution.The Spice Cayenne data accelerator exits Beta with significant reliability and performance improvements:
\npartition_by: [col1, col2] using hierarchical path-like keys (e.g., 2025/10/15).UPDATE operations.DELETE + UPSERT sequences could produce duplicate primary keys across protected snapshots. Improved upsert handling during runtime restarts.Apache DataFusion has been upgraded to v52.2.0, bringing significant performance improvements, new query features, and enhanced extensibility.
\nPerformance Improvements:
\nCASE Expressions: Lookup-table-based evaluation for certain CASE expressions avoids repeated evaluation, accelerating common ETL patternsMIN/MAX Aggregate Dynamic Filters: Queries with MIN/MAX aggregates now create dynamic filters during scan to prune files and rows as tighter bounds are discovered during executionMajor Features:
\nDESC queries on ASC-sorted filesTableProvider supports DELETE and UPDATE: New hooks for DELETE and UPDATE statements in the TableProvider trait, enabling Iceberg and Cayenne connectors to implement SQL DELETE and UPDATE operationsRelationPlanner API for extending SQL planning for FROM clauses, enabling support for vendor-specific SQL dialectsSQL Schema Management: Spice now supports CREATE TABLE and DROP TABLE DDL operations for Iceberg and Cayenne catalogs via FlightSQL and the /v1/sql API. DML validation has been updated for catalog-level writability.
Lakehouse-Style Data Management: New DuckLake catalog and data connector enable lakehouse-style data management with DuckDB as the metadata catalog and object storage for data files. DuckLake provides ACID transactions, time travel, and schema evolution on top of Parquet files.
\nGoogle Cloud Storage Support: New Google Cloud Storage data connector enables federated queries against data stored in GCS buckets, with Iceberg table support.
\nUnified Single-Binary Experience: The Spice CLI has been completely rewritten from Go to Rust, eliminating the Go dependency and providing a single spice binary built from the same codebase as spiced. This improves startup performance, reduces distribution size, and ensures consistent behavior between CLI and runtime.
Key Features:
\nspice query Command: Interactive REPL for async queries via the /v1/queries API with multi-line SQL input, spinner progress indicator, Ctrl+C cancellation, and partial query ID matching--output=json Flag: Machine-readable JSON output for CLI commands, enabling scripting and automationspice login --output: New output modes (env, json, keychain) for flexible credential managementspice cloud metrics: New command for Spice Cloud deployment metricsLocal LLM/ML model inference (via mistral.rs) is now included in the default Spice build. The separate models build variant has been removed. This simplifies installation and ensures all users have access to local AI inference capabilities.
The /v1/datasets and /v1/models APIs now return structured error information when a component is in an Error state. The ?status=true query parameter must be passed to retrieve the real-time component status, including the error state and details. Previously, the status field only indicated Error with no further detail. Now, two new fields are included when ?status=true is specified:
error: A structured object with category, type, and code fields for programmatic error handling (e.g. { \"category\": \"dataset\", \"type\": \"auth\", \"code\": \"dataset.auth\" }).error_message: A human-readable description of why the component entered an error state.These fields are only present when ?status=true is passed and the component is in an error state.
Example /v1/datasets?status=true response:
[
{
\"from\": \"postgres:syncs\",
\"name\": \"daily_journal\",
\"replication_enabled\": false,
\"acceleration_enabled\": true,
\"status\": \"Ready\"
},
{
\"from\": \"databricks:hive_metastore.default.messages\",
\"name\": \"messages\",
\"replication_enabled\": false,
\"acceleration_enabled\": true,
\"status\": \"Error\",
\"error\": {
\"category\": \"dataset\",
\"type\": \"auth\",
\"code\": \"dataset.auth\"
},
\"error_message\": \"Unable to authenticate with datasource credentials\"
}
]
The spice datasets and spice models CLI commands now include an ERROR column that displays the error message for any component in an error state.
| Dependency | Version |
|---|---|
| Ballista | v52.0.0 |
| DuckDB | v1.4.4 |
| delta_kernel | v0.18.2 |
| mistral.rs | v0.7.0 (candle fork removed, now uses candle 0.9.2 from crates.io) |
| Turso (libsql) | v0.4.4 |
| Vortex | Upgraded with CASE-WHEN support |
| AWS SDK | Multiple crates updated + APN user-agent support |
v2, and spice init generates spicepod.yaml files with version: v2 by default while maintaining backward compatibility for existing v1 spicepods./v1/responses endpoint with rate limiting support.snowflake_private_key parameter for key-pair authentication.rows_written, bytes_written, and dataset_acceleration_size_bytes metrics for acceleration refresh ingestion.grpc+tls:// endpoints with custom CA certificate support.serde_yaml with an internal YAML implementation.Spicepod v2 introduces configuration improvements while maintaining backward compatibility with v1. Existing v1 spicepods continue to work β deprecated fields are automatically migrated at load time.
Version support:
\n| Version | Status |
|---|---|
v2 | Default. Used by spice init. |
v1 | Supported. Deprecated fields auto-migrate. |
v1beta1 | Removed. No longer accepted. |
Configuration changes:
\n| v1 (deprecated) | v2 (preferred) | Notes |
|---|---|---|
runtime.results_cache | runtime.caching.sql_results | All fields migrate automatically. cache_max_size β max_size. |
runtime.memory_limit | runtime.query.memory_limit | Auto-migrated. query.memory_limit takes priority if both set. |
runtime.temp_directory | runtime.query.temp_directory | Auto-migrated. query.temp_directory takes priority if both set. |
dataset.invalid_type_action | dataset.unsupported_type_action | Auto-migrated. v2 adds a new string variant. |
New v2 fields:
\nruntime.ready_state β Controls when the runtime reports ready (on_load default, or on_registration).runtime.flight.do_put_rate_limit_enabled β Enable/disable FlightSQL DoPut rate limiting (default: true).runtime.query.spill_compression β Compression for query spill files (e.g., lz4_frame).runtime.scheduler.partition_management β Configure partition assignment interval, limits, and timeouts for distributed mode.runtime.caching.sql_results.stale_while_revalidate_ttl β Serve stale cached results while revalidating in the background.runtime.caching.sql_results.encoding β Cache entry compression (e.g., zstd).catalog.access: read_write_create β New access mode for catalogs that support DDL operations.Migration note: When both the deprecated v1 field and its v2 equivalent are set, the v2 field takes priority.
\nmodels build variant has been removed. Local LLM inference is now always included.v2: New spicepods created with spice init now default to version: v2. Existing v1 spicepods remain supported, and v1beta1 is no longer accepted.accelerated_refresh metrics renamed to acceleration_refresh for consistency. last_refresh_time gauge renamed to include milliseconds unit.ResultsCache replaced with SQLResultsCacheConfig in configuration.partitioned_write_flush_threshold renamed to partitioned_write_flush_threshold_rows./v1/search API now always returns an array in matches, even for single results./v1/responses endpoint.New and updated Spice Cookbook recipes:
\nThe Spice Cookbook includes 88 recipes to help you get started with Spice quickly and easily.
\nTo upgrade to v2.0.0-rc.1, use one of the following methods:
\nCLI:
\nspice upgrade
Homebrew:
\nbrew upgrade spiceai/spiceai/spice
Docker:
\nPull the spiceai/spiceai:2.0.0-rc.1 image:
docker pull spiceai/spiceai:2.0.0-rc.1
For available tags, see DockerHub.
\nHelm:
\nhelm repo update
helm upgrade spiceai spiceai/spiceai --version 2.0.0-rc.1
AWS Marketplace:
\nSpice is available in the AWS Marketplace.
\ntime_interval and fix snapshots_creation_policy: on_change by @sgrebnov in #9026release/1.11.0-rc.2 by @krinart in #9016formula_1 and codebase_community in bird-bench by @Jeadie in #9000PartitionManagementTask for scheduler to update accelerated table partition assignments by @Jeadie in #9378CayenneDataSink by @sgrebnov in #9402aws_lc_rs since jsonwebtoken upgrade by @Jeadie in #9426spice-cloud-client crate and spice cloud metrics --app <app-name>. by @Jeadie in #9480lint-rust-fix support targeted packages and features by @Jeadie in #9511pr-builds --tag <TAG> for build_and_release.yml by @Jeadie in #9507--output flag to spice login with env/json/keychain modes by @Jeadie in #9541AcceleratedTable::scan use FederatedTable::scan when ClusterRole::Scheduler by @Jeadie in #9550Refresh::refresh_sql more robust to alterations over time. by @Jeadie in #9549AWS_ENDPOINT_URL when empty by @krinart in #9575spice query CLI response deserialization by @phillipleblanc in #9588Full Changelog: https://github.com/spiceai/spiceai/compare/v1.11.2...v2.0.0-rc.1
", "url": "https://spiceai.org/releases/v2.0-rc.1", "title": "Spice v2.0-rc.1 (Mar 4, 2026)", "summary": "Announcing the release of Spice v2.0-rc.1! π", "date_modified": "2026-03-04T00:00:00.000Z", "author": { "name": "Sergei Grebnov", "url": "https://github.com/sgrebnov" }, "tags": [ "release", "distributed-query", "cayenne", "datafusion", "ducklake", "gcp", "Rust", "data-connector" ] }, { "id": "https://spiceai.org/releases/v1.11.1", "content_html": "Announcing the release of Spice v1.11.1! π οΈ
\nv1.11.1 is a patch release improving Spice Cayenne accelerator reliability and performance, enhancing DynamoDB Streams and HTTP data connectors, and fixing issues in Federated Task History and FlightSQL.
\nThis release includes stability and performance fixes for the Spice Cayenne accelerator:
\nRoaringBitmap. Deletion scans now use Vortex-native streaming with filter pushdown and project only row indices, achieving zero data I/O for delete operations.constraints and on_conflict configurations are now automatically inferred from federated table metadata, enabling datasets like DynamoDB to work without explicitly defining primary_key in the Spicepod.DELETE operations on partitioned Cayenne tables failed.lag_exceeds_shard_retention_behavior parameter with values error (default), ready_before_load, or ready_after_load.response_status column (UInt16). 4xx responses (e.g., 404 Not Found) are treated as valid queryable data and cached normally. 5xx responses are retried with backoff, returned to the user, but excluded from the cache to prevent transient server errors from polluting cached results.SnapshotManager operations and general snapshot reliability improvements.Schema error: project index out of bounds errors in cluster mode.No breaking changes.
\nNo major cookbook updates. The Spice Cookbook includes 86 recipes to help you get started with Spice quickly and easily.
\nTo upgrade to v1.11.1, use one of the following methods:
\nCLI:
\nspice upgrade
Homebrew:
\nbrew upgrade spiceai/spiceai/spice
Docker:
\nPull the spiceai/spiceai:1.11.1 image:
docker pull spiceai/spiceai:1.11.1
For available tags, see DockerHub.
\nHelm:
\nhelm repo update
helm upgrade spiceai spiceai/spiceai --version 1.11.1
AWS Marketplace:
\nSpice is available in the AWS Marketplace.
\nconstraints/on_conflict in Cayenne Acceleration by @krinart in #9335SchemaCastScanExec before applying changes in process_upsert_batch by @krinart in #9297Announcing the release of Spice v1.11.0-stable! β‘
\nIn Spice v1.11.0, Spice Cayenne reaches Beta status with acceleration snapshots, Key-based deletion vectors, and Amazon S3 Express One Zone support. DataFusion has been upgraded to v51 along with Arrow v57.2, and iceberg-rust v0.8.0. v1.11 adds several DynamoDB & DynamoDB Streams improvements such as JSON nesting, and adds significant improvements to Distributed Query with active-active schedulers and mTLS for enterprise-grade high-availability and secure cluster communication.
\nThis release also adds new SMB, NFS, and ScyllaDB Data Connectors (Alpha), Prepared Statements with full SDK support (gospice, spice-rs, spice-dotnet, spice-java, spice.js, and spicepy), Google LLM Support for expanded AI inference capabilities, and significant improvements to caching, observability, and Hash Indexing for Arrow Acceleration.
\nSpice Cayenne has been promoted to Beta status with acceleration snapshots support and numerous performance and stability improvements.
\nKey Enhancements:
\nImproved Reliability:
\nFuturesUnordered reentrant drop crashescayenne_file_path locationFor more details, refer to the Cayenne Documentation.
\nApache DataFusion has been upgraded to v51, bringing significant performance improvements, new SQL features, and enhanced observability.
\n
Performance Improvements:
\nCASE Expression Evaluation: Expressions now short-circuit earlier, reuse partial results, and avoid unnecessary scattering, speeding up common ETL patternsNew SQL Features:
\n|> syntax for inline transformsDESCRIBE <query>: Returns the schema of any query without executing itparam => value syntax for scalar, aggregate, and window functionsSUM, AVG, and MIN/MAXExample pipe operator:
\nSELECT * FROM t
|> WHERE a > 10
|> ORDER BY b
|> LIMIT 5;
Improved Observability:
\nEXPLAIN ANALYZE Metrics: New metrics including output_bytes, selectivity for filters, reduction_factor for aggregates, and detailed timing breakdownsApache Arrow has been upgraded to v57.2, bringing major performance improvements and new capabilities.
\n![\"Arrow](\"https://arrow.apache.org/img/rust-parquet-metadata/results.png\"){kind=icon}
Key Features:
\nGEOMETRY and GEOGRAPHY) with GeospatialStatisticsarrow-avro Crate: Efficient conversion between Apache Avro and Arrow RecordBatches with projection pushdown and vectorized execution supportHigh Availability Clusters: Spice now supports running multiple active schedulers in an active/active configuration for production deployments. This eliminates the scheduler as a single point of failure and enables graceful handling of node failures.
\nExample HA configuration:
\nruntime:
scheduler:
state_location: s3://my-bucket/spice-cluster
params:
region: us-east-1
mTLS Verification: Cluster communication between scheduler and executors now supports mutual TLS verification for enhanced security.
\nCredential Propagation: S3, ABFS, and GCS credentials are now automatically propagated to executors in cluster mode, enabling access to cloud storage across the distributed query cluster.
\nImproved Resilience:
\n--scheduler-address is providedFor more details, refer to the Distributed Query Documentation.
\nSpice has been upgraded to iceberg-rust v0.8.0, bringing improved Iceberg table support.
\nKey Features:
\nINSERT INTO Partitioned Tables: DataFusion integration now supports inserting data into partitioned Iceberg tablesupdate_table and register_table for SQL catalogupdate_table for S3 Tables catalog_file column support, and Date32 type supportAcceleration snapshots enable point-in-time recovery and data versioning for accelerated datasets. Snapshots capture the state of accelerated data at specific points, allowing for fast bootstrap recovery and rollback capabilities.
\nKey Features:
\nExample configuration:
\ndatasets:
- from: s3://my-bucket/data.parquet
name: my_dataset
acceleration:
enabled: true
engine: cayenne
mode: file
snapshots: enabled
snapshots_trigger: time_interval
snapshots_trigger_threshold: 1h
snapshots_creation_policy: on_changed
Snapshots API and CLI: New API endpoints and CLI commands for managing snapshots programmatically.
\nCLI Commands:
\n# List all snapshots for a dataset
spice acceleration snapshots taxi_trips
# Get details of a specific snapshot
spice acceleration snapshot taxi_trips 3
# Set the current snapshot for rollback (requires runtime restart)
spice acceleration set-snapshot taxi_trips 2
HTTP API Endpoints:
\n| Method | Endpoint | Description |
|---|---|---|
| GET | /v1/datasets/{dataset}/acceleration/snapshots | List all snapshots for a dataset |
| GET | /v1/datasets/{dataset}/acceleration/snapshots/{id} | Get details of a specific snapshot |
| POST | /v1/datasets/{dataset}/acceleration/snapshots/current | Set the current snapshot for rollback |
For more details, refer to the Acceleration Snapshots Documentation.
\nThe Caching Acceleration Mode introduced in v1.10.0 has received significant performance optimizations and reliability fixes in this release.
\nPerformance Optimizations:
\nReliability Fixes:
\nfetched_at in their projection would always result in cache misses, even when cached data was available.SELECT * queries without filters now return cached data directly without unnecessary filtering overhead.For more details, refer to the Caching Acceleration Mode Documentation.
\nImproved Query Performance and Security: Spice now supports prepared statements, enabling parameterized queries that improve both performance through query plan caching and security by preventing SQL injection attacks.
\nKey Features:
\nSDK Support:
\n| SDK | Support | Min Version | Method |
|---|---|---|---|
| gospice (Go) | β Full | v8.0.0+ | SqlWithParams() with typed constructors (Int32Param, StringParam, TimestampParam, etc.) |
| spice-rs (Rust) | β Full | v3.0.0+ | query_with_params() with RecordBatch parameters |
| spice-dotnet (.NET) | β Full | v0.3.0+ | QueryWithParams() with typed parameter builders |
| spice-java (Java) | β Full | v0.5.0+ | queryWithParams() with typed Param constructors (Param.int64(), Param.string(), etc.) |
| spice.js (JavaScript) | β Full | v3.1.0+ | query() with parameterized query support |
| spicepy (Python) | β Full | v3.1.0+ | query() with parameterized query support |
Example (Go):
\nimport \"github.com/spiceai/gospice/v8\"
client, _ := spice.NewClient()
defer client.Close()
// Parameterized query with typed parameters
results, _ := client.SqlWithParams(ctx,
\"SELECT * FROM products WHERE price > $1 AND category = $2\",
spice.Float64Param(10.0),
spice.StringParam(\"electronics\"),
)
Example (Java):
\nimport ai.spice.SpiceClient;
import ai.spice.Param;
import org.apache.arrow.adbc.core.ArrowReader;
try (SpiceClient client = new SpiceClient()) {
// With automatic type inference
ArrowReader reader = client.queryWithParams(
\"SELECT * FROM products WHERE price > $1 AND category = $2\",
10.0, \"electronics\");
// With explicit typed parameters
ArrowReader reader = client.queryWithParams(
\"SELECT * FROM products WHERE price > $1 AND category = $2\",
Param.float64(10.0),
Param.string(\"electronics\"));
}
For more details, refer to the Parameterized Queries Documentation.
\nParameterized Query Support for Java: The Spice Java SDK v0.5.0 introduces parameterized queries using ADBC (Arrow Database Connectivity), providing a safer and more efficient way to execute queries with dynamic parameters.
\nKey Features:
\ndouble β Float64, String β Utf8)Param class with typed factory methods (Param.int64(), Param.string(), Param.decimal128(), etc.) for precise control over Arrow typesExample:
\nimport ai.spice.SpiceClient;
import ai.spice.Param;
try (SpiceClient client = new SpiceClient()) {
// With automatic type inference
ArrowReader reader = client.queryWithParams(
\"SELECT * FROM taxi_trips WHERE trip_distance > $1 LIMIT 10\",
5.0);
// With explicit typed parameters for precise control
ArrowReader reader = client.queryWithParams(
\"SELECT * FROM orders WHERE order_id = $1 AND amount >= $2\",
Param.int64(12345),
Param.decimal128(new BigDecimal(\"99.99\"), 10, 2));
}
Maven:
\n<dependency>
<groupId>ai.spice</groupId>
<artifactId>spiceai</artifactId>
<version>0.5.0</version>
</dependency>
For more details, refer to the Spice Java SDK Repository.
\nExpanded AI Provider Support: Spice now supports Google embedding and chat models via the Google AI provider, expanding the available LLM options for AI inference workloads alongside existing providers like OpenAI, Anthropic, and AWS Bedrock.
\nKey Features:
\nExample spicepod.yaml configuration:
\nmodels:
- from: google:gemini-2.0-flash
name: gemini
params:
google_api_key: ${secrets:GOOGLE_API_KEY}
embeddings:
- from: google:text-embedding-004
name: google_embeddings
params:
google_api_key: ${secrets:GOOGLE_API_KEY}
For more details, refer to the Google LLM Documentation (see docs PR #1286).
\nQuery data sources directly via URL in SQL without prior dataset registration. Supports S3, Azure Blob Storage, and HTTP/HTTPS URLs with automatic format detection and partition inference.
\nSupported Patterns:
\nSELECT * FROM 's3://bucket/data.parquet'SELECT * FROM 's3://bucket/data/'SELECT * FROM 's3://bucket/year=*/month=*/data.parquet'Key Features:
\nExample with hive partitioning:
\n-- Partitions are automatically inferred from paths
SELECT * FROM 's3://bucket/data/' WHERE year = '2024' AND month = '01'
Enable via spicepod.yml:
\nruntime:
params:
url_tables: enabled
New asynchronous query APIs for long-running queries in cluster mode:
\n/v1/queries endpoint: Submit queries and retrieve results asynchronouslyUnified Telemetry Endpoint: OTel metrics ingestion has been consolidated to the Flight port (50051), simplifying deployment by removing the separate OTel port (50052). The push-based metrics exporter continues to support integration with OpenTelemetry collectors.
\n\n\nNote: This is a breaking change. Update your configurations if you were using the dedicated OTel port 50052. Internal cluster communication now uses port 50052 exclusively.
\n
Enhanced Dashboards: Updated Grafana and Datadog example dashboards with:
\nAdditional Histogram Buckets: Added more buckets to histogram metrics for better latency distribution visibility.
\nFor more details, refer to the Monitoring Documentation.
\nArrow-based accelerations now support hash indexing for faster point lookups on equality predicates. Hash indexes provide O(1) average-case lookup performance for columns with high cardinality.
\nFeatures:
\nExample configuration:
\ndatasets:
- from: postgres:users
name: users
acceleration:
enabled: true
engine: arrow
primary_key: user_id
indexes:
'(tenant_id, user_id)': unique # Composite hash index
For more details, refer to the Hash Index Documentation.
\nNetwork-Attached Storage Connectors: New data connectors for SMB (Server Message Block) and NFS (Network File System) protocols enable direct federated queries against network-attached storage without requiring data movement to cloud object stores.
\nKey Features:
\nExample spicepod.yaml configuration:
\ndatasets:
# SMB share
- from: smb://fileserver/share/data.parquet
name: smb_data
params:
smb_username: ${secrets:SMB_USER}
smb_password: ${secrets:SMB_PASS}
# NFS export
- from: nfs://nfsserver/export/data.parquet
name: nfs_data
For more details, refer to the Data Connectors Documentation.
\nA new data connector for ScyllaDB, the high-performance NoSQL database compatible with Apache Cassandra. Query ScyllaDB tables directly or accelerate them for faster analytics.
\nExample configuration:
\ndatasets:
- from: scylladb:my_keyspace.my_table
name: scylla_data
acceleration:
enabled: true
engine: duckdb
For more details, refer to the ScyllaDB Data Connector Documentation.
\nTLS Connection Support: Fixed TLS connection issues when using grpc+tls:// scheme with Flight SQL endpoints. Added support for custom CA certificate files via the new flightsql_tls_ca_certificate_file parameter.
spice cloud CLI commands for cloud deployment managementsnapshots_create_interval for periodic acceleration snapshots independent of refresh cyclescaching refresh mode, enabling multi-layer acceleration where a persistent cache feeds a fast in-memory cacheon_refresh_sort_columns parameter for DuckDB enables data ordering after writesdynamodb_auth: iam_role parametertemp_directory parameter for shuffle filesDistributeFileScanOptimizer plansmax_message_size configuration for Flight serviceOTel ingestion has been moved to the Flight port (50051), removing the separate OTel port 50052. Port 50052 is now used exclusively for internal cluster communication. Update your configurations if you were using the dedicated OTel port.
\nDistributed query cluster mode now requires mTLS for secure communication between cluster nodes. This is a security enhancement to prevent unauthorized nodes from joining the cluster and accessing secrets.
\nMigration Steps:
\nspice cluster tls init and spice cluster tls add--node-mtls-* arguments--allow-insecure-connections to opt out of mTLSRenamed CLI Arguments:
\n| Old Name | New Name |
|---|---|
--cluster-mode | --role |
--cluster-ca-certificate-file | --node-mtls-ca-certificate-file |
--cluster-certificate-file | --node-mtls-certificate-file |
--cluster-key-file | --node-mtls-key-file |
--cluster-address | --node-bind-address |
--cluster-advertise-address | --node-advertise-address |
--cluster-scheduler-url | --scheduler-address |
Removed CLI Arguments:
\n--cluster-api-key: Replaced by mTLS authenticationNew ScyllaDB Data Connector Recipe: New recipe demonstrating how to use the ScyllaDB Data Connector. See ScyllaDB Data Connector Recipe for details.
\nNew SMB Data Connector Recipe: New recipe demonstrating how to use the SMB Data Connector. See SMB Data Connector Recipe for details.
\nThe Spice Cookbook includes 86 recipes to help you get started with Spice quickly and easily.
\nTo upgrade to v1.11.0, use one of the following methods:
\nCLI:
\nspice upgrade
Homebrew:
\nbrew upgrade spiceai/spiceai/spice
Docker:
\nPull the spiceai/spiceai:1.11.0 image:
docker pull spiceai/spiceai:1.11.0
For available tags, see DockerHub.
\nHelm:
\nhelm repo update
helm upgrade spiceai spiceai/spiceai --version 1.11.0
AWS Marketplace:
\nSpice is available in the AWS Marketplace.
\nchunking crate. by @Jeadie in #8431aggregate_pushdown_optimization -> optimizer_duckdb_aggregate_pushdown by @ewgenius in #8485runtime-datafusion by @Jeadie in #8490make lint-rust-fix with make lint-rust by @Jeadie in #8499captured_plan outputs to be captured even if they would be filtered out otherwise by @phillipleblanc in #8501Datafusion::Plan over Datafusion::Internal for user-facing search errors. by @Jeadie in #8484clippy::too_many_lines lint by @phillipleblanc in #8549spice cluster tls commands by @phillipleblanc in #8550spice cluster tls add certificates by @phillipleblanc in #8554DistributeFileScanOptimizer plans by @mach-kernel in #8581(Deprecated) labels to deprecated spice sql params by @krinart in #8588security.protocol is set to PLAINTEXT by @krinart in #8587rstest for llms integration tests by @Jeadie in #8566spicepod-validator cargo build from build-dev target by @Jeadie in #8684model.datasets to allowlist on tools by @Jeadie in #8714tools from auto-defaults by @Jeadie in #8725text-to-sql dispatch in testoperator by @Jeadie in #8705ghcr.io. by @Jeadie in #8732testoperator_dispatch.yaml by @Jeadie in #8740testoperator run text-to-sql. by @Jeadie in #8745table_allowlist for table sampling and NSQL by @Jeadie in #8789auth/iam_role_source to DynamoDB connector by @krinart in #8808UnionProjectionPushdownOptimizer - Schema change during transform_down breaks parent nodes by @phillipleblanc in #8831release/1.11 branch by @ewgenius in #8829testoperator run text-to-sql metrics from LogicalPlan by @Jeadie in #8895BootstrapStatus + Snapshot bootstrapping parallelization by @krinart in #8926cayenne_metadata_dir to known params by @sgrebnov in #8933snapshots_creation_policy param by @krinart in #8954data_components arrow::indexed::test_primary_key_value_matches_batch by @Jeadie in #8962testoperator run text-to-sql. by @Jeadie in #8927Chat::as_sql for xAI anthropic by @Jeadie in #8957duckdb_file_path in search integration test by @Jeadie in #8972on_refresh_sort_columns for post-write data ordering (initial version) by @sgrebnov in #8964.spice/data in search integration tests by @Jeadie in #8992snapshots_creation_policy by @krinart in #9003time_interval and fix snapshots_creation_policy: on_change by @sgrebnov in #9026release/1.11.0-rc.2 by @krinart in #9016testoperator by @krinart in #90541.11.0-rc.3 release notes by @krinart in #9082formula_1 and codebase_community in bird-bench by @Jeadie in #9000spice query command for async queries REPL by @phillipleblanc in #9057dataset_acceleration_size_bytes metric by @krinart in #9062sql_query beneath tool_use::sample_data. by @Jeadie in #9043snowflake_private_key parameter by @ewgenius in #9085DistributeFileScanOptimizer and UnionProjectionPushdownOptimizer & set target_partitions dynamically based on cluster capacity by @phillipleblanc in #9100Announcing the release of Spice v1.11.0-rc.3! β
\nv1.11.0-rc.3 is a patch release that includes improvements to Hash Indexing for Arrow Acceleration and fixes for TLS connections with Flight SQL endpoints.
\nArrow-based accelerations now support hash indexing for faster point lookups on equality predicates. Hash indexes provide O(1) average-case lookup performance for columns with high cardinality.
\nFeatures:
\nExample configuration:
\ndatasets:
- from: postgres:users
name: users
acceleration:
enabled: true
engine: arrow
primary_key: user_id
indexes:
'(tenant_id, user_id)': unique # Composite hash index
For more details, refer to the Hash Index Documentation.
\nTLS Connection Support: Fixed TLS connection issues when using grpc+tls:// scheme with Flight SQL endpoints. Added support for custom CA certificate files via the new flightsql_tls_ca_certificate_file parameter.
No breaking changes.
\nNo major cookbook updates.
\nThe Spice Cookbook includes 86 recipes to help you get started with Spice quickly and easily.
\nTo upgrade to v1.11.0-rc.3, use one of the following methods:
\nCLI:
\nspice upgrade
Homebrew:
\nbrew upgrade spiceai/spiceai/spice
Docker:
\nPull the spiceai/spiceai:v1.11.0-rc.3 image:
docker pull spiceai/spiceai:v1.11.0-rc.3
For available tags, see DockerHub.
\nHelm:
\nhelm repo update
helm upgrade spiceai spiceai/spiceai --version 1.11.0-rc.3
AWS Marketplace:
\nSpice is available in the AWS Marketplace.
\nAnnouncing the release of Spice v1.11.0-rc.2! β
\nv1.11.0-rc.2 is the second release candidate for advanced test of v1.11. It brings Spice Cayenne to Beta status with acceleration snapshots support, a new ScyllaDB Data Connector, upgrades to DataFusion v51, Arrow 57.2, and iceberg-rust v0.8.0. It includes significant improvements to distributed query, caching, and observability.
\nSpice Cayenne has been promoted to Beta status with acceleration snapshots support and numerous stability improvements.
\nImproved Reliability:
\nFuturesUnordered reentrant drop crashescayenne_file_path locationExample configuration with snapshots:
\ndatasets:
- from: s3://my-bucket/data.parquet
name: my_dataset
acceleration:
enabled: true
engine: cayenne
mode: file
Apache DataFusion has been upgraded to v51, bringing significant performance improvements, new SQL features, and enhanced observability.
\n
Performance Improvements:
\nCASE Expression Evaluation: Expressions now short-circuit earlier, reuse partial results, and avoid unnecessary scattering, speeding up common ETL patternsNew SQL Features:
\n|> syntax for inline transformsDESCRIBE <query>: Returns the schema of any query without executing itparam => value syntax for scalar, aggregate, and window functionsSUM, AVG, and MIN/MAXExample pipe operator:
\nSELECT * FROM t
|> WHERE a > 10
|> ORDER BY b
|> LIMIT 5;
Improved Observability:
\nEXPLAIN ANALYZE Metrics: New metrics including output_bytes, selectivity for filters, reduction_factor for aggregates, and detailed timing breakdownsSpice has been upgraded to Apache Arrow Rust 57.2.0, bringing major performance improvements and new capabilities.
\n
Key Features:
\nGEOMETRY and GEOGRAPHY) with GeospatialStatisticsarrow-avro Crate: Efficient conversion between Apache Avro and Arrow RecordBatches with projection pushdown and vectorized execution supportSpice has been upgraded to iceberg-rust v0.8.0, bringing improved Iceberg table support.
\nKey Features:
\nINSERT INTO Partitioned Tables: DataFusion integration now supports inserting data into partitioned Iceberg tablesupdate_table and register_table for SQL catalogupdate_table for S3 Tables catalog_file column support, and Date32 type supportAcceleration snapshots enable point-in-time recovery and data versioning for accelerated datasets. Snapshots capture the state of accelerated data at specific points, allowing for fast bootstrap recovery and rollback capabilities.
\nKey Feature Improvements in v1.11:
\nExample configuration:
\ndatasets:
- from: s3://my-bucket/data.parquet
name: my_dataset
acceleration:
enabled: true
engine: cayenne
mode: file
snapshots: enabled
snapshots_trigger: time_interval
snapshots_trigger_threshold: 1h
snapshots_creation_policy: on_changed
Snapshots API and CLI: New API endpoints and CLI commands for managing snapshots programmatically. List, create, and restore snapshots directly from the command line or via HTTP.
\nFor more details, refer to the Acceleration Snapshots Documentation.
\nA new data connector for ScyllaDB, the high-performance NoSQL database compatible with Apache Cassandra. Query ScyllaDB tables directly or accelerate them for faster analytics.
\nExample configuration:
\ndatasets:
- from: scylladb:my_keyspace.my_table
name: scylla_data
acceleration:
enabled: true
engine: duckdb
For more details, refer to the ScyllaDB Data Connector Documentation.
\nmTLS Verification: Cluster communication between scheduler and executors now supports mutual TLS verification for enhanced security.
\nCredential Propagation: Azure and GCS credentials are now automatically propagated to executors in cluster mode, enabling access to cloud storage across the distributed query cluster.
\nImproved Resilience:
\n--scheduler-address is providedFor more details, refer to the Distributed Query Documentation.
\nThe Caching Acceleration Mode introduced in v1.10.0 has received significant performance optimizations and reliability fixes in this release.
\nPerformance Optimizations:
\nReliability Fixes:
\nfetched_at in their projection would always result in cache misses, even when cached data was available.SELECT * queries without filters now return cached data directly without unnecessary filtering overhead.For more details, refer to the Caching Acceleration Mode Documentation.
\nQuery data sources directly via URL in SQL without prior dataset registration. Supports S3, Azure Blob Storage, and HTTP/HTTPS URLs with automatic format detection and partition inference.
\nSupported Patterns:
\nSELECT * FROM 's3://bucket/data.parquet'SELECT * FROM 's3://bucket/data/'SELECT * FROM 's3://bucket/year=*/month=*/data.parquet'Key Features:
\nExample with hive partitioning:
\n-- Partitions are automatically inferred from paths
SELECT * FROM 's3://bucket/data/' WHERE year = '2024' AND month = '01'
Enable via spicepod.yml:
\nruntime:
params:
url_tables: enabled
New asynchronous query APIs for long-running queries in cluster mode:
\n/v1/queries endpoint: Submit queries and retrieve results asynchronouslyEnhanced Dashboards: Updated Grafana and Datadog example dashboards with:
\nAdditional Histogram Buckets: Added more buckets to histogram metrics for better latency distribution visibility.
\nFor more details, refer to the Monitoring Documentation.
\non_refresh_sort_columns parameter for DuckDB enables data ordering after writesNo breaking changes.
\nNew ScyllaDB Data Connector Recipe: New recipe demonstrating how to use ScyllaDB Data Connector. See ScyllaDB Data Connector Recipe for details.
\nNew SMB Data Connector Recipe: New recipe demonstrating how to use ScyllaDB Data Connector. See SMB Data Connector Recipe for details.
\nThe Spice Cookbook includes 86 recipes to help you get started with Spice quickly and easily.
\nTo upgrade to v1.11.0-rc.2, use one of the following methods:
\nCLI:
\nspice upgrade
Homebrew:
\nbrew upgrade spiceai/spiceai/spice
Docker:
\nPull the spiceai/spiceai:v1.11.0-rc.2 image:
docker pull spiceai/spiceai:v1.11.0-rc.2
For available tags, see DockerHub.
\nHelm:
\nhelm repo update
helm upgrade spiceai spiceai/spiceai
AWS Marketplace:
\nSpice is available in the AWS Marketplace.
\ntable_allowlist for table sampling and NSQL by @Jeadie in #8789auth/iam_role_source to DynamoDB connector by @krinart in #8808UnionProjectionPushdownOptimizer - Schema change during transform_down breaks parent nodes by @phillipleblanc in #8831release/1.11 branch by @ewgenius in #8829testoperator run text-to-sql metrics from LogicalPlan by @Jeadie in #8895BootstrapStatus + Snapshot bootstrapping parallelization by @krinart in #8926cayenne_metadata_dir to known params by @sgrebnov in #8933snapshots_creation_policy param by @krinart in #8954data_components arrow::indexed::test_primary_key_value_matches_batch by @Jeadie in #8962testoperator run text-to-sql. by @Jeadie in #8927Chat::as_sql for xAI anthropic by @Jeadie in #8957duckdb_file_path in search integration test by @Jeadie in #8972on_refresh_sort_columns for post-write data ordering (initial version) by @sgrebnov in #8964.spice/data in search integration tests by @Jeadie in #8992snapshots_creation_policy by @krinart in #9003Announcing the release of Spice v1.11.0-rc.1! β
\nv1.11.0-rc.1 is the first release candidate for early testing of v1.11 features including Distributed Query with mTLS for enterprise-grade secure cluster communication, new SMB and NFS Data Connectors for direct network-attached storage access, Prepared Statements for improved query performance and security, Cayenne Accelerator Enhancements with Key-based deletion vectors and Amazon S3 Express One Zone support, Google LLM Support for expanded AI inference capabilities, and Spice Java SDK v0.5.0 with parameterized query support.
\nEnterprise-Grade Secure Cluster Communication: Distributed query cluster mode now enables mutual TLS (mTLS) by default for secure communication between schedulers and executors. Internal cluster communication includes highly privileged RPC calls like fetching Spicepod configuration and expanding secrets. mTLS ensures only authenticated nodes can join the cluster and access sensitive data.
\nKey Features:
\nspice cluster tls init and spice cluster tls add commands for generating CA certificates and node certificates with proper SANs (Subject Alternative Names)--role, --scheduler-address, --node-mtls-*) with --scheduler-address implying --role executorSchedulerGrpcServer, ClusterService) are isolated on port 50052 with mTLS enforced--allow-insecure-connections flag to disable mTLS requirement for local development and testingQuick Start:
\n# Generate certificates for development
spice cluster tls init
spice cluster tls add scheduler1
spice cluster tls add executor1
# Start scheduler
spiced --role scheduler \\
--node-mtls-ca-certificate-file ca.crt \\
--node-mtls-certificate-file scheduler1.crt \\
--node-mtls-key-file scheduler1.key
# Start executor
spiced --role executor \\
--scheduler-address https://scheduler1:50052 \\
--node-mtls-ca-certificate-file ca.crt \\
--node-mtls-certificate-file executor1.crt \\
--node-mtls-key-file executor1.key
For more details, refer to the Distributed Query Documentation.
\nNetwork-Attached Storage Connectors: New data connectors for SMB (Server Message Block) and NFS (Network File System) protocols enable direct federated queries against network-attached storage without requiring data movement to cloud object stores.
\nKey Features:
\nExample spicepod.yaml configuration:
\ndatasets:
# SMB share
- from: smb://fileserver/share/data.parquet
name: smb_data
params:
smb_username: ${secrets:SMB_USER}
smb_password: ${secrets:SMB_PASS}
# NFS export
- from: nfs://nfsserver/export/data.parquet
name: nfs_data
For more details, refer to the Data Connectors Documentation.
\nImproved Query Performance and Security: Spice now supports prepared statements, enabling parameterized queries that improve both performance through query plan caching and security by preventing SQL injection attacks.
\nKey Features:
\nSDK Support:
\n| SDK | Support | Min Version | Method |
|---|---|---|---|
| gospice (Go) | β Full | v8.0.0+ | SqlWithParams() with typed constructors (Int32Param, StringParam, TimestampParam, etc.) |
| spice-rs (Rust) | β Full | v3.0.0+ | query_with_params() with RecordBatch parameters |
| spice-dotnet (.NET) | β Not yet | - | Coming soon |
| spice-java (Java) | β Full | v0.5.0+ | queryWithParams() with typed Param constructors (Param.int64(), Param.string(), etc.) |
| spice.js (JavaScript) | β Not yet | - | Coming soon |
| spicepy (Python) | β Not yet | - | Coming soon |
Example (Go):
\nimport \"github.com/spiceai/gospice/v8\"
client, _ := spice.NewClient()
defer client.Close()
// Parameterized query with typed parameters
results, _ := client.SqlWithParams(ctx,
\"SELECT * FROM products WHERE price > $1 AND category = $2\",
spice.Float64Param(10.0),
spice.StringParam(\"electronics\"),
)
Example (Java):
\nimport ai.spice.SpiceClient;
import ai.spice.Param;
import org.apache.arrow.adbc.core.ArrowReader;
try (SpiceClient client = new SpiceClient()) {
// With automatic type inference
ArrowReader reader = client.queryWithParams(
\"SELECT * FROM products WHERE price > $1 AND category = $2\",
10.0, \"electronics\");
// With explicit typed parameters
ArrowReader reader = client.queryWithParams(
\"SELECT * FROM products WHERE price > $1 AND category = $2\",
Param.float64(10.0),
Param.string(\"electronics\"));
}
For more details, refer to the Parameterized Queries Documentation.
\nThe Spice Cayenne data accelerator has been improved with several key enhancements:
\nExample spicepod.yaml configuration:
\ndatasets:
- from: s3://my-bucket/data.parquet
name: fast_data
acceleration:
enabled: true
engine: cayenne
mode: file
params:
# Use S3 Express One Zone for data files
cayenne_s3express_bucket: my-express-bucket--usw2-az1--x-s3
For more details, refer to the Cayenne Documentation.
\nExpanded AI Provider Support: Spice now supports Google embedding and chat models via the Google AI provider, expanding the available LLM options for AI inference workloads alongside existing providers like OpenAI, Anthropic, and AWS Bedrock.
\nKey Features:
\nExample spicepod.yaml configuration:
\nmodels:
- from: google:gemini-2.0-flash
name: gemini
params:
google_api_key: ${secrets:GOOGLE_API_KEY}
embeddings:
- from: google:text-embedding-004
name: google_embeddings
params:
google_api_key: ${secrets:GOOGLE_API_KEY}
For more details, refer to the Google LLM Documentation (see docs PR #1286).
\nParameterized Query Support for Java: The Spice Java SDK v0.5.0 introduces parameterized queries using ADBC (Arrow Database Connectivity), providing a safer and more efficient way to execute queries with dynamic parameters.
\nKey Features:
\ndouble β Float64, String β Utf8)Param class with typed factory methods (Param.int64(), Param.string(), Param.decimal128(), etc.) for precise control over Arrow typesExample:
\nimport ai.spice.SpiceClient;
import ai.spice.Param;
try (SpiceClient client = new SpiceClient()) {
// With automatic type inference
ArrowReader reader = client.queryWithParams(
\"SELECT * FROM taxi_trips WHERE trip_distance > $1 LIMIT 10\",
5.0);
// With explicit typed parameters for precise control
ArrowReader reader = client.queryWithParams(
\"SELECT * FROM orders WHERE order_id = $1 AND amount >= $2\",
Param.int64(12345),
Param.decimal128(new BigDecimal(\"99.99\"), 10, 2));
}
Maven:
\n<dependency>
<groupId>ai.spice</groupId>
<artifactId>spiceai</artifactId>
<version>0.5.0</version>
</dependency>
For more details, refer to the Spice Java SDK Repository.
\nUnified Telemetry Endpoint: OTel metrics ingestion has been consolidated to the Flight port (50051), simplifying deployment by removing the separate OTel port (50052). The push-based metrics exporter continues to support integration with OpenTelemetry collectors.
\n\n\nNote: This is a breaking change. Update your configurations if you were using the dedicated OTel port 50052. Internal cluster communication now uses port 50052 exclusively.
\n
spice cloud CLI commands for cloud deployment managementsnapshots_create_interval for periodic acceleration snapshots independent of refresh cyclescaching refresh mode, enabling multi-layer acceleration where a persistent cache feeds a fast in-memory cachedynamodb_auth: iam_role parametertemp_directory parameter for shuffle filesDistributeFileScanOptimizer plansmax_message_size configuration for Flight serviceOTel ingestion has been moved to the Flight port (50051), removing the separate OTel port 50052. Port 50052 is now used exclusively for internal cluster communication. Update your configurations if you were using the dedicated OTel port.
\nDistributed query cluster mode now requires mTLS for secure communication between cluster nodes. This is a security enhancement to prevent unauthorized nodes from joining the cluster and accessing secrets.
\nMigration Steps:
\nspice cluster tls init and spice cluster tls add--node-mtls-* arguments--allow-insecure-connections to opt out of mTLSRenamed CLI Arguments:
\n| Old Name | New Name |
|---|---|
--cluster-mode | --role |
--cluster-ca-certificate-file | --node-mtls-ca-certificate-file |
--cluster-certificate-file | --node-mtls-certificate-file |
--cluster-key-file | --node-mtls-key-file |
--cluster-address | --node-bind-address |
--cluster-advertise-address | --node-advertise-address |
--cluster-scheduler-url | --scheduler-address |
Removed CLI Arguments:
\n--cluster-api-key: Replaced by mTLS authenticationNo major cookbook updates.
\nThe Spice Cookbook includes 84 recipes to help you get started with Spice quickly and easily.
\nTo try v1.11.0-rc.1, use one of the following methods:
\nCLI:
\nspice upgrade --version 1.11.0-rc.1
Homebrew:
\nbrew upgrade spiceai/spiceai/spice
Docker:
\nPull the spiceai/spiceai:1.11.0-rc.1 image:
docker pull spiceai/spiceai:1.11.0-rc.1
For available tags, see DockerHub.
\nHelm:
\nhelm repo update
helm upgrade spiceai spiceai/spiceai --version 1.11.0-rc.1
AWS Marketplace:
\nπ Spice is available in the AWS Marketplace!
\nchunking crate. by @Jeadie in #8431aggregate_pushdown_optimization -> optimizer_duckdb_aggregate_pushdown by @ewgenius in #8485runtime-datafusion by @Jeadie in #8490make lint-rust-fix with make lint-rust by @Jeadie in #8499captured_plan outputs to be captured even if they would be filtered out otherwise by @phillipleblanc in #8501model.datasets to allowlist on tools by @Jeadie in #8714tools from auto-defaults by @Jeadie in #8725text-to-sql dispatch in testoperator by @Jeadie in #8705ghcr.io. by @Jeadie in #8732testoperator_dispatch.yaml by @Jeadie in #8740testoperator run text-to-sql. by @Jeadie in #8745auth/iam_role_source to DynamoDB connector by @krinart in #8808Announcing the release of Spice v1.10.4! π οΈ
\nv1.10.4 is a patch release with fixes for Kafka/Debezium batch commits, ABFSS URL support for Azure Data Lake Storage Gen2, and improved column projection handling for location metadata columns.
\nabfss:// URL prefix for Azure Data Lake Storage Gen2, in addition to the existing abfs:// prefix. The abfss scheme indicates secure (TLS) connections to ADLS Gen2.SELECT location, day, size FROM dataset). Queries that specified columns in a different order than the schema would fail with \"column types must match schema types\" errors.No breaking changes.
\nNo major cookbook updates.
\nThe Spice Cookbook includes 84 recipes to help you get started with Spice quickly and easily.
\nTo upgrade to v1.10.4, use one of the following methods:
\nCLI:
\nspice upgrade
Homebrew:
\nbrew upgrade spiceai/spiceai/spice
Docker:
\nPull the spiceai/spiceai:1.10.4 image:
docker pull spiceai/spiceai:1.10.4
For available tags, see DockerHub.
\nHelm:
\nhelm repo update
helm upgrade spiceai spiceai/spiceai
AWS Marketplace:
\nπ Spice is now available in the AWS Marketplace!
\nAnnouncing the release of Spice v1.10.3! π
\nv1.10.3 is a patch release with improved startup reliability, fixes for Azure BlobFS versioned containers, S3 custom endpoint query resolution, and a fix for the OpenAI Responses API.
\nlocation predicates on datasets using s3_endpoint and s3_region parameters now correctly route to the configured endpoint instead of defaulting to AWS S3./v1/responses) was not working correctly.No breaking changes.
\nNo major cookbook updates.
\nThe Spice Cookbook includes 84 recipes to help you get started with Spice quickly and easily.
\nTo upgrade to v1.10.3, use one of the following methods:
\nCLI:
\nspice upgrade
Homebrew:
\nbrew upgrade spiceai/spiceai/spice
Docker:
\nPull the spiceai/spiceai:1.10.3 image:
docker pull spiceai/spiceai:1.10.3
For available tags, see DockerHub.
\nHelm:
\nhelm repo update
helm upgrade spiceai spiceai/spiceai
AWS Marketplace:
\nπ Spice is now available in the AWS Marketplace!
\nAnnouncing the release of Spice v1.10.2! π₯
\nv1.10.2 introduces Tiered Caching Acceleration with Localpod for multi-layer acceleration architectures, Periodic Acceleration Snapshots with configurable intervals, DynamoDB JSON Nesting for column consolidation, and Kafka/Debezium Batching for faster data ingestion. This release also includes fixes for SQLite accelerator decimal/date handling and real-time status reporting for the /v1/datasets and /v1/models API endpoints.
Multi-Layer Acceleration Architecture: The Localpod connector now supports caching refresh mode, enabling tiered acceleration where a persistent cache (e.g., file-mode DuckDB) feeds a fast in-memory cache (e.g., Arrow, memory-mode DuckDB).
Key Features:
\nExample spicepod.yaml configuration:
datasets:
# Parent: persistent file-mode cache
- from: https://api.example.com
name: api_cache
acceleration:
enabled: true
refresh_mode: caching
engine: duckdb
mode: file
# Child: fast in-memory cache fed by parent
- from: localpod:api_cache
name: api_cache_memory
acceleration:
enabled: true
refresh_mode: caching
engine: arrow
mode: memory
For more details, refer to the Localpod Data Connector Documentation.
\nConfigurable Snapshot Intervals: A new snapshots_create_interval parameter enables periodic snapshot creation for accelerated datasets across all refresh modes. This provides better control over snapshot frequency and ensures consistent recovery points for accelerated data.
Example spicepod.yaml configuration:
datasets:
- from: s3://my-bucket/data.parquet
name: my_data
acceleration:
enabled: true
engine: duckdb
mode: file
refresh_mode: caching
snapshots: enabled
params:
snapshots_create_interval: 60s # Write a snapshot every 60 seconds
For more details, refer to the Data Acceleration Documentation.
\nConsolidate Columns into JSON: The DynamoDB Data Connector now supports consolidating columns into a single JSON column using the json_object: \"*\" metadata option. This is useful when only a few columns are needed as discrete fields while the rest can be accessed as nested JSON.
Example spicepod.yaml configuration:
datasets:
- from: dynamodb:my_table
name: my_table
columns:
- name: PK
- name: SK
- name: data_json
metadata:
json_object: '*' # Captures all other columns as JSON
Example Output: Given a DynamoDB table with columns PK, SK, name, email, and status, the resulting table schema consolidates all non-specified columns into the data_json column:
| PK | SK | data_json |
|---|---|---|
| pk_1 | sort_1 | {\"name\": \"Alice\", \"email\": \"alice@example.com\", \"status\": \"active\"} |
| pk_2 | sort_2 | {\"name\": \"Bob\", \"email\": \"bob@example.com\", \"status\": \"inactive\"} |
For more details, refer to the DynamoDB JSON Nesting Documentation.
\nFaster Data Ingestion: Configure message batching for Kafka and Debezium connectors to improve data ingestion throughput. Batching reduces processing overhead by grouping multiple messages together before insertion.
\nKey Features:
\nExample spicepod.yaml configuration:
datasets:
- from: debezium:kafka-server.public.my_table
name: my_table
params:
batch_max_size: 10000 # Max records per batch (default: 10000)
batch_max_duration: 1s # Max wait time per batch (default: 1s)
For more details, refer to the Kafka Data Connector Documentation and Debezium Data Connector Documentation.
\n/v1/datasets and /v1/models API endpoints.security.protocol is set to PLAINTEXT.No breaking changes.
\nNew Cayenne Data Accelerator Recipe: New recipe demonstrating how to accelerate a local copy of the taxi trips dataset using Cayenne as the data accelerator engine. See Cayenne Data Accelerator Recipe for details.
\nNew Dataset Partitioning Recipe: New recipe demonstrating how to partition accelerated datasets to improve query performance. See Dataset Partitioning for details.
\nThe Spice Cookbook includes 84 recipes to help you get started with Spice quickly and easily.
\nTo upgrade to v1.10.2, use one of the following methods:
\nCLI:
\nspice upgrade
Homebrew:
\nbrew upgrade spiceai/spiceai/spice
Docker:
\nPull the spiceai/spiceai:1.10.2 image:
docker pull spiceai/spiceai:1.10.2
For available tags, see DockerHub.
\nHelm:
\nhelm repo update
helm upgrade spiceai spiceai/spiceai
AWS Marketplace:
\nπ Spice is now available in the AWS Marketplace!
\nsecurity.protocol is set to PLAINTEXT by @krinart in #8587clippy::too_many_lines lint by @phillipleblanc in #8549Announcing the release of Spice v1.10.1! π
\nv1.10.1 is a patch release with Cayenne accelerator improvements including configurable compression strategies and improved partition ID handling, isolated refresh runtime for better query API responsiveness, and security hardening. In addition, the GO SDK, gospice v8 has been released.
\nSeveral improvements and bug fixes for the Cayenne data accelerator:
\ncayenne_compression_strategy parameter enables choosing between zstd for compact storage or btrblocks for encoding-efficient compression.Example spicepod.yaml configuration:
datasets:
- from: s3://my-bucket/data.parquet
name: my_dataset
acceleration:
enabled: true
engine: cayenne
mode: file
params:
cayenne_compression_strategy: zstd # or btrblocks (default)
For more details, refer to the Cayenne Data Accelerator Documentation.
\nRefresh tasks now run on a separate Tokio runtime isolated from the main query API. This prevents long-running or resource-intensive refresh operations from impacting query latency and ensures the /health endpoint remains responsive during heavy refresh workloads.
Multiple security improvements have been implemented:
\n/v1/spicepods endpoint now returns summarized information instead of full spicepod.yaml representations, preventing potential sensitive information leakage.The Spice Go SDK has been upgraded to v8 with a cleaner API, parameterized queries, and health check methods: gospice v8.0.0.
\nKey Features:
\nSql() and SqlWithParams() methods with more intuitive naming.Decimal128Param, TimestampParam, and more.IsSpiceHealthy() and IsSpiceReady() methods for instance monitoring.Example usage with a local Spice runtime:
\nimport \"github.com/spiceai/gospice/v8\"
// Initialize client for local runtime
spice := gospice.NewSpiceClient()
defer spice.Close()
if err := spice.Init(
gospice.WithFlightAddress(\"grpc://localhost:50051\"),
); err != nil {
panic(err)
}
// Parameterized query (safe from SQL injection)
reader, err := spice.SqlWithParams(
ctx,
\"SELECT * FROM users WHERE id = $1 AND created_at > $2\",
userId,
startTime,
)
Upgrade:
\ngo get github.com/spiceai/gospice/v8@v8.0.0
For more details, refer to the Go SDK Documentation.
\n/v1/spicepods no longer returns the full spicepod.yaml JSON representation. A summary is returned instead. See #8404.No major cookbook updates.
\nThe Spice Cookbook includes 82+ recipes to help you get started with Spice quickly and easily.
\nTo upgrade to v1.10.1, use one of the following methods:
\nCLI:
\nspice upgrade
Homebrew:
\nbrew upgrade spiceai/spiceai/spice
Docker:
\nPull the spiceai/spiceai:1.10.1 image:
docker pull spiceai/spiceai:1.10.1
For available tags, see DockerHub.
\nHelm:
\nhelm repo update
helm upgrade spiceai spiceai/spiceai
AWS Marketplace:
\nπ Spice is now available in the AWS Marketplace!
\nruntime-datafusion by @Jeadie in #8490DistributeFileScanOptimizer plans by @mach-kernel in #8581Announcing the release of Spice v1.10.0! β‘
\nSpice v1.10.0 introduces a new Caching Acceleration Mode with stale-while-revalidate (SWR) semantics for disk-persisted, low-latency queries with background refresh. This release also adds the TinyLFU eviction policy for the SQL results cache, a preview of the DynamoDB Streams connector for real-time CDC, S3 location predicate pruning for faster partitioned queries, improved distributed query execution, and multiple security hardening improvements.
\nLow-Latency Queries with Background Refresh: This release introduces a new caching acceleration mode that implements the stale-while-revalidate (SWR) pattern. Queries return cached results immediately while data refreshes asynchronously in the background, eliminating query latency spikes during refresh cycles. Cached data persists to disk using DuckDB, SQLite, or Cayenne file modes.
Key Features:
\nrefresh_check_interval to balance freshness and source load\n\nRecommendation: Use retention configuration with caching acceleration to ensure stale data is cleaned up over time.
\n
Example spicepod.yaml configuration:
\ndatasets:
- from: http://localhost:7400
name: cached_data
time_column: fetched_at
acceleration:
enabled: true
engine: duckdb
mode: file # Persist cache to disk
refresh_mode: caching
refresh_check_interval: 10m
retention_check_enabled: true
retention_period: 24h
retention_check_interval: 1h
For more details, refer to the Data Acceleration Documentation.
\nHigher Cache Hit Rates for SQL Results Cache: A new TinyLFU cache eviction policy is now available for the SQL results cache. TinyLFU is a probabilistic cache admission policy that maintains higher hit rates than LRU while keeping memory usage predictable, making it ideal for workloads with varying query frequency patterns.
\nExample spicepod.yaml configuration:
\nruntime:
caching:
sql_results:
enabled: true
eviction_policy: tiny_lfu # default: lru
For more details, refer to the Caching Documentation and the Moka TinyLFU Documentation for details of the algorithm.
\nReal-Time Change Data Capture for DynamoDB: The DynamoDB connector now integrates with DynamoDB Streams for real-time change data capture (CDC). This enables continuous synchronization of DynamoDB table changes into Spice for real-time query, search, and LLM-inference.
\nKey Features:
\nrefresh_mode: changes to incrementally update accelerated datasets\n\nNote: DynamoDB Streams must be enabled on your DynamoDB table. This feature is in preview.
\n
Example spicepod.yaml configuration:
\ndatasets:
- from: dynamodb:my_table
name: orders_stream
acceleration:
enabled: true
refresh_mode: changes # Enable Streams capture
For more details, refer to the DynamoDB Connector Documentation.
\nSpice can now push metrics to an OpenTelemetry collector, enabling integration with platforms such as Jaeger, New Relic, Honeycomb, and other OpenTelemetry-compatible backends.
\nKey Features:
\nExample spicepod.yaml configuration for gRPC:
\nruntime:
telemetry:
enabled: true
otel_exporter:
endpoint: 'localhost:4317'
push_interval: '30s'
For more details, refer to the Observability & Monitoring Documentation.
\nS3 Location Predicate Pruning: The S3 data connector now supports location-based predicate pruning, dramatically reducing data scanned by pushing down location filter predicates to S3 listing operations. For partitioned datasets (e.g., year=2025/month=12/), Spice now skips listing irrelevant partitions entirely, significantly reducing query latency and S3 API costs.
AWS S3 Tables Write Support: Full read/write capability for AWS S3 Tables, enabling direct integration with AWS's managed table format for S3. Use standard SQL INSERT INTO to write data.
For more details, refer to the S3 Data Connector Documentation and Glue Data Connector Documentation.
\nDistributed query planning and execution have been significantly improved:
\nDistributeFileScanOptimizer improvements for faster planning with large datasetsFor more details, refer to the Distributed Query Documentation.
\nSearch capabilities have been improved with several performance and reliability enhancements:
\nlist_vectors calls for better performanceIndexerExec now properly handles limit pushdown for more efficient searchesFor more details, refer to the Search Documentation.
\nMultiple security improvements have been implemented:
\ntop_n_sample order_by clause parsing to prevent injection attacks.clear history command in the REPL to fully clear persisted historyNo breaking changes.
\nNo major cookbook updates.
\nThe Spice Cookbook includes 82 recipes to help you get started with Spice quickly and easily.
\nTo upgrade to v1.10.0, use one of the following methods:
\nCLI:
\nspice upgrade
Homebrew:
\nbrew upgrade spiceai/spiceai/spice
Docker:
\nPull the spiceai/spiceai:1.10.0 image:
docker pull spiceai/spiceai:1.10.0
For available tags, see DockerHub.
\nHelm:
\nhelm repo update
helm upgrade spiceai spiceai/spiceai
AWS Marketplace:
\nπ Spice is now available in the AWS Marketplace!
\n.clear history by @lukekim in #8254sort_columns on insert by @lukekim in #8091caching by @lukekim in #8237dynamodb-streams crate by @krinart in #8283list_vectors (or equivalent) when indexing into VectorIndexs by @Jeadie in #8282IndexerExec to properly handle limit pushdown by @sgrebnov in #8366stale-while-revalidate pattern. by @phillipleblanc in #8365on_conflict verification more robust by @sgrebnov in #8375.vscode/launch.json by @sgrebnov in #8415google-genai by @Jeadie in #8390with_retention_data to dispatch by @sgrebnov in #8430caching_stale_if_error to caching accelerator; fix multiple upstream requests during SWR; fix Arrow accelerator by @phillipleblanc in #8425snapshots_trigger_threshold for DynamoDB Streams by @krinart in #8437duckdb-partitioned query override by @sgrebnov in #8446aggregate_pushdown_optimization -> optimizer_duckdb_aggregate_pushdown by @ewgenius in #8485Announcing the release of Spice v1.10.0-rc.1! β‘
\nv1.10.0-rc1 is a release candidate for early testing of v1.10 features including an all new caching acceleration mode, tiny_lfu caching policy, a new DynamoDB Streams connector (Preview), improvements to the DynamoDB connector, faster distributed query execution, S3 connector improvements, and security hardening for v1.10.0-stable.
This release introduces a new caching acceleration mode that implements the stale-while-revalidate (SWR) pattern using Data Accelerators such as DuckDB or Cayenne, enabling queries to return file-persisted cached results immediately while asynchronously refreshing data in the background. Combined with the new TinyLFU cache eviction policy, Spice can now maintain higher cache hit rates while keeping memory usage predictable.
Key Features:
\nExample Spicepod.yml configuration:
\nruntime:
caching:
sql_results:
enabled: true
eviction_policy: tiny_lfu # default lru
datasets:
- from: s3://my-bucket/data.parquet
name: cached_data
acceleration:
enabled: true
engine: duckdb
mode: file # Persist cache to disk
refresh_mode: caching
refresh_check_interval: 10m
For more details, refer to the Data Acceleration Documentation and Caching Documentation.
\nDynamoDB Connector now integrates with DynamoDB Streams which enables real-time streaming with support for both table bootstrapping and continuous change data capture (CDC). This connector automatically detects changes in DynamoDB tables and streams them into Spice for real-time query, search, and LLM-inference.
\nKey Features:
\nExample Spicepod.yml configuration:
\ndatasets:
- from: dynamodb:my_table
name: orders_stream
acceleration:
enabled: true
refresh_mode: changes
For more details, refer to the DynamoDB Connector Documentation.
\nThe Cayenne data accelerator now supports:
\nExample Spicepod.yml configuration:
\ndatasets:
- from: s3://my-bucket/data.parquet
name: sorted_data
acceleration:
enabled: true
engine: cayenne
mode: file_create
params:
sort_columns: timestamp,region
For more details, refer to the Cayenne Documentation.
\nS3 Location Predicate Pruning: The S3 data connector now supports location-based predicate pruning, dramatically reducing data scanned by pushing down predicates to S3 listing operations. This optimization is especially effective for partitioned datasets stored in S3.
\nAWS S3 Tables Write Support: Full read/write capability for AWS S3 Tables, enabling fast integration with AWS's table format for S3.
\nFor more details, refer to the S3 Tables Data Connector Documentation and Glue Data Connection Documentation.
\nDistributed query planning and execution have been significantly improved:
\nDistributeFileScanOptimizer improvements for faster planning with large datasetsFor more details, refer to the Distributed Query Documentation.
\nSearch capabilities have been improved with several performance and reliability enhancements:
\nlist_vectors calls for better performanceIndexerExec now properly handles limit pushdown for more efficient searchesFor more details, refer to the Search Documentation.
\nMultiple security improvements have been implemented:
\ntop_n_sample order_by parsing.clear history command in the REPL to fully clear persisted historyNo breaking changes.
\nNo major cookbook updates. The Spice Cookbook still offers 82+ recipes to help you prototype quickly.
\nTo try v1.10.0-rc1, use one of the following methods:
\nCLI:
\nspice upgrade --version 1.10.0-rc1
Homebrew:
\nbrew upgrade spiceai/spiceai/spice
Docker:
\nPull the spiceai/spiceai:1.10.0-rc1 image:
docker pull spiceai/spiceai:1.10.0-rc1
For available tags, see DockerHub.
\nHelm:
\nhelm repo update
helm upgrade spiceai spiceai/spiceai --version 1.10.0-rc1
AWS Marketplace:
\nπ Spice is available in the AWS Marketplace.
\n.clear history by @lukekim in #8254sort_columns on insert by @lukekim in #8091caching by @lukekim in #8237dynamodb-streams crate by @krinart in #8283list_vectors (or equivalent) when indexing into VectorIndexs by @Jeadie in #8282IndexerExec to properly handle limit pushdown by @sgrebnov in #8366stale-while-revalidate pattern. by @phillipleblanc in #8365Announcing the release of Spice v1.9.1!π₯
\nv1.9.1 introduces Amazon Bedrock Nova 2 Multimodal embeddings support with high-dimensional vectors up to 3,072 dimensions and purpose-optimized embeddings for semantic search and retrieval operations, DynamoDB timestamp filter pushdown for more efficient append-mode acceleration with configurable time formatting, HTTP Data Connector health probe configuration for improved endpoint validation reliability, and Spice .NET SDK v0.2 with expanded .NET version support and updated gRPC libraries. This release focuses on bug fixes, stability, and performance improvements.
\nSpice now supports the Amazon Nova 2 Multimodal embeddings models via the Bedrock models provider, enabling high-quality text embeddings for semantic search and vector similarity operations. The Nova embeddings model offers configurable dimensions and advanced features like truncation modes and embedding purpose optimization.
\nKey Features:
\nSTART, END, or NONE)GENERIC_INDEX, GENERIC_RETRIEVAL, or CLASSIFICATION)Example spicepod.yml configuration:
embeddings:
- from: bedrock:amazon.nova-2-multimodal-embeddings-v1:0
name: nova_embeddings
params:
dimensions: '3072' # Required: Output dimensions
truncation_mode: START # Optional: START, END, or NONE (default: NONE)
embedding_purpose: GENERIC_RETRIEVAL # Optional. GENERIC_INDEX is default
For more details on the embedding parameters and configuration options, refer to the Amazon Nova Embeddings Documentation and the Spice Embeddings Documentation.
\nThe DynamoDB Data Connector now supports timestamp filter pushdown, enabling more efficient append-mode acceleration refreshes by pushing timestamp filters directly to DynamoDB queries. Since DynamoDB stores timestamps as strings rather than native datetime types, this feature includes configurable timestamp formatting to ensure correct parsing and filtering.
\nKey Features:
\nExample spicepod.yml configuration:
datasets:
- from: dynamodb:sales
name: sales
time_column: created_at
time_format: timestamptz
params:
time_format: 2006-01-02T15:04:05.000Z07:00
acceleration:
enabled: true
engine: duckdb
refresh_mode: append
For more details, refer to the DynamoDB Data Connector Documentation.
\nThe HTTP Data Connector now supports configurable health probe paths for endpoint validation. Instead of using a random non-existent path, the system can now validate endpoints using a user-specified path, improving flexibility and reliability for health checks.
\nExample spicepod.yml configuration:
datasets:
- from: https://api.tvmaze.com
name: tvmaze
params:
file_format: json
health_probe: /health-check
For more details, refer to the HTTP Data Connector Documentation.
\nThe Spice .NET SDK has been upgraded with expanded .NET version support, custom User-Agent configuration, and updated gRPC libraries: spice-dotnet v0.2.0. The SDK is available on NuGet.
\nKey Features:
\nnetstandard for improved performance and reliabilityUpgrade Example:
\ndotnet add package SpiceAI --version 0.2.0
For more details, refer to the .NET SDK Documentation.
\ntrust_dns_resolver to hickory_resolver for improved DNS resolution reliability.runtime.* schema queries from being sent to the scheduler, improving performance for metadata queries.No breaking changes.
\nNo major cookbook updates.
\nThe Spice Cookbook includes 82 recipes to help you get started with Spice quickly and easily.
\nTo upgrade to v1.9.1, use one of the following methods:
\nCLI:
\nspice upgrade
Homebrew:
\nbrew upgrade spiceai/spiceai/spice
Docker:
\nPull the spiceai/spiceai:1.9.1 image:
docker pull spiceai/spiceai:1.9.1
For available tags, see DockerHub.
\nHelm:
\nhelm repo update
helm upgrade spiceai spiceai/spiceai
AWS Marketplace:
\nπ Spice is now available in the AWS Marketplace!
\namazon.nova-2-multimodal-embeddings-v1:0 by @Jeadie in #8225runtime.* schema queries to scheduler by @mach-kernel in #8271Announcing the release of Spice v1.9.0-stable! πΆ
\nv1.9.0-stable introduces Spice Cayenne, a new high-performance data accelerator built on the Vortex columnar format that delivers better than DuckDB performance without single-file scaling limitations, and a preview of Multi-Node Distributed Query based on Apache Ballista. v1.9.0 also upgrades to DataFusion v50, DuckDB v1.4.2, and Delta-Kernel v0.16 for even higher query performance, expands search capabilities with full-text search on views and multi-column embeddings, and delivers many additional features and improvements.
\nIntroducing Cayenne: SQL as an Acceleration Format: A new high-performance Data Accelerator that simplifies multi-file data acceleration by using an embedded database (SQLite) for metadata while storing data in the Vortex columnar format, a Linux Foundation project. Cayenne delivers query and ingestion performance better than DuckDB's file-based acceleration without DuckDB's memory overhead and the scaling challenges of single DuckDB files.
\nCayenne uses SQLite to manage acceleration metadata (schemas, snapshots, statistics, file tracking) through simple SQL transactions, while storing data in Vortex's compressed columnar format. This architecture provides:
\nKey Features:
\nExample Spicepod.yml configuration:
\ndatasets:
- from: s3:my_table
name: accelerated_data_30d
acceleration:
enabled: true
engine: cayenne
mode: file
refresh_mode: append
retention_sql: DELETE FROM accelerated_data WHERE created_at < NOW() - INTERVAL '30 days'
Note, the Cayenne Data Accelerator is in Beta with limitations.
\nFor more details, refer to the Cayenne Documentation, the Vortex project, and the DuckLake announcement that partly inspired this design.
\nApache Ballista Integration: Spice now supports distributed query execution based on Apache Ballista, enabling distributed queries across multiple executor nodes for improved performance on large datasets. This feature is in preview in v1.9.0.
\nArchitecture:
\nA distributed Spice cluster consists of:
\nGetting Started:
\nStart a scheduler instance using an existing Spicepod. The scheduler is the only spiced instance that needs to be configured:
\n# Start scheduler (note the flight bind address override if you want it reachable outside localhost)
spiced --cluster-mode scheduler --flight 0.0.0.0:50051
Start one or more executors configured with the scheduler's flight URI:
\n# Start executor (automatically selects a free port if 50051 is taken)
spiced --cluster-mode executor --scheduler-url spiced://localhost:50051
Query Execution:
\nQueries run through the scheduler will now show a distributed_plan in EXPLAIN output, demonstrating how the query is distributed across executor nodes:
EXPLAIN SELECT count(id) FROM my_dataset;
Current Limitations:
\nFor more details, refer to the Distributed Query Documentation.
\nSpice.ai is built on the Apache DataFusion query engine. The v50 release brings significant performance improvements and enhanced reliability:
\nPerformance Improvements π:
\nDynamic Filter Pushdown: Enhanced dynamic filter pushdown for custom ExecutionPlans, ensuring filters propagate correctly through all physical operators for improved query performance.
Partition Pruning: Expanded partition pruning support ensures that unnecessary partitions are skipped when filters are not used, reducing data scanning overhead and improving query execution times.
\nApache Spark Compatible Functions: Added support for Spark-compatible functions including array, bit_get/bit_count, bitmap_count, crc32/sha1, date_add/date_sub, if, last_day, like/ilike, luhn_check, mod/pmod, next_day, parse_url, rint, and width_bucket.
Bug Fixes & Reliability: Resolved issues with partition name validation and empty execution plans when vector index lists are empty. Fixed timestamp support for partition expressions, enabling better partitioning for time-series data.
\nSee the Apache DataFusion 50.0.3 Release for more details.
\nDuckDB v1.4.2: DuckDB has been upgraded to v1.4.2, which includes several performance optimizations.
\nComposite ART Index Support: DuckDB in Spice now supports composite (multi-column) Adaptive Radix Tree (ART) indexes for accelerated table scans. When queries filter on multiple columns fully covered by a composite index, the optimizer automatically uses index scans instead of full table scans, delivering significant performance improvements for selective queries.
\nExample configuration:
\ndatasets:
- from: file://data.parquet
name: sales
acceleration:
enabled: true
engine: duckdb
indexes:
'(region, product_id)': enabled
Performance example with composite index on 7.5M rows:
\nSELECT * FROM sales WHERE region = 'US' AND product_id = 12345;
-- Without index: 0.282s
-- With composite index (region, product_id): 0.037s
-- Performance improvement: 7.6x faster with composite index
DuckDB Intermediate Materialization: Queries with indexes now use intermediate materialization (WITH ... AS MATERIALIZED) to leverage faster index scans. Currently supported for non-federated queries (query_federation: disabled) against a single table with indexes only. When predicates cover more columns than the index, the optimizer rewrites queries to first materialize index-filtered results, then apply remaining predicates. This optimization can deliver significant performance improvements for selective queries.
Example configuration:
\ndatasets:
- from: file://sales_data.parquet
name: sales
acceleration:
enabled: true
engine: duckdb
mode: file
params:
query_federation: disabled # Required currently for intermediate materialization
indexes:
'(region, product_id)': enabled
Performance example:
\n-- Query with indexed columns (region, product_id) plus additional filter (amount)
SELECT * FROM sales
WHERE region = 'US' AND product_id = 12345 AND amount > 1000;
-- Optimized execution time: 0.031s (with intermediate materialization)
-- Standard execution time: 0.108s (without optimization)
-- Performance improvement: ~3.5x faster
The optimizer automatically rewrites the query to:
\nWITH _intermediate_materialize AS MATERIALIZED (
SELECT * FROM sales WHERE region = 'US' AND product_id = 12345
)
SELECT * FROM _intermediate_materialize WHERE amount > 1000;
Parquet Buffering for Partitioned Writes: DuckDB partitioned writes in table mode now support Parquet buffering, reducing memory usage and improving write performance for large datasets.
\nRetention SQL on Refresh Commit: DuckDB accelerations now support running retention SQL on refresh commit, enabling automatic data cleanup and lifecycle management during refresh operations.
\nUTC Timezone for DuckDB: DuckDB now uses UTC as the default timezone, ensuring consistent behavior for time-based queries across different environments.
\nExample Spicepod.yml configuration:
\ndatasets:
- from: s3://my_bucket/large_table/
name: partitioned_data
acceleration:
enabled: true
engine: duckdb
mode: file
retention:
sql: DELETE FROM partitioned_data WHERE event_time < NOW() - INTERVAL '7 days'
For more details, refer to the DuckDB Data Accelerator Documentation.
\nQuerying endpoints as tables: The HTTP/HTTPS Data Connectors now supports querying HTTP endpoints directly as tables in SQL queries with dynamic filters. This feature transforms REST APIs into queryable data sources, making it easy to integrate external service data.
\nQuery HTTP endpoint that returns structured data (JSON, CSV, etc.) as if it were a database table
\nConfigurable retry logic, timeouts, and POST request support for more complex API interactions
\nExample Spicepod.yml configuration:
\ndatasets:
- from: https://api.tvmaze.com
name: tvmaze
params:
file_format: json
max_retries: 3
client_timeout: 10s
allowed_request_paths: /search/people
request_query_filters: enabled
request_body_filters: enabled
Example SQL query:
\nSELECT request_path, request_query, content
FROM tvmaze
WHERE request_path = '/search/people' and request_query = 'q=michael'
LIMIT 10;
If a request_body is supplied it will be posted to the endpoint:
Example SQL query:
\nSELECT request_path, request_query, content
FROM tvmaze
WHERE request_path = '/search/people' and request_query = 'q=michael' and request_body = '{\"name\": \"michael\"}'
LIMIT 10;
HTTP endpoints can be accelerated using refresh_sql:
datasets:
- from: https://api.tvmaze.com
name: tvmaze
params:
file_format: json
allowed_request_paths: /search/people
request_query_filters: enabled
request_body_filters: enabled
acceleration:
enabled: true
refresh_mode: full
refresh_sql: |
SELECT request_path, request_query, content
FROM tvmaze
WHERE request_path = '/search/people'
AND request_query IN ('q=michael', 'q=luke')
For more details, refer to the HTTP Data Connector Documentation.
\nImproved Query Performance: The DynamoDB Data Connector now includes improved filter handling for edge cases, parallel scan support for faster data ingestion, and better error handling for misconfigured queries. These improvements enable more reliable and performant access to DynamoDB data.
\nExample Spicepod.yml configuration:
\ndatasets:
- from: dynamodb:my_table
name: ddb_data
params:
scan_segments: 10 # Default `auto` which calculates optimal segments based on number of rows
For more details, refer to the DynamoDB Data Connector Documentation.
\nS3 Versioning Support: Spice now supports S3 Versioning for all connectors using object-store (S3, Delta Lake, etc.), ensuring range reads over versioned files are atomically correct. When S3 versioning is enabled, Spice automatically tracks version IDs during file discovery and uses them for all subsequent range reads, preventing inconsistencies from concurrent file modifications.
\nCurrent limitations:
\nS3 Single-File Refresh Skipping: Spice now optimizes S3 single-file dataset refreshes by caching file metadata (ETag, Version ID, size, timestamp) and skipping unnecessary data fetches when the underlying file hasn't changed. This optimization dramatically reduces bandwidth usage and improves refresh performance for scenarios where data doesn't change frequently. The feature is enabled by default for accelerated S3 single-file datasets and includes metrics tracking for skipped refreshes.
\nExample configuration:
\ndatasets:
- from: s3://my-bucket/data.parquet
name: s3_data
acceleration:
enabled: true
engine: duckdb
refresh_check_interval: 10s
When the file's metadata hasn't changed between refresh checks, Spice will skip the data fetch entirely, logging:
\nSkipping refresh for dataset 's3_data': file metadata unchanged
For more details, refer to the S3 Data Connector Documentation.
\nFull-Text Search on Views: Full-text search indexes are now supported on views, enabling advanced search scenarios over pre-aggregated or transformed data. This extends the power of Spice's search capabilities beyond base datasets.
\nMulti-Column Embeddings on Views: Views now support embedding columns, enabling vector search and semantic retrieval on view data. This is useful for search over aggregated or joined datasets.
\nVector Engines on Views: Vector search engines are now available for views, enabling similarity search over complex queries and transformations.
\nExample Spicepod.yml configuration:
\nviews:
- name: aggregated_reviews
sql: SELECT review_id, review_text FROM reviews WHERE rating > 4
embeddings:
- column: review_text
model: openai:text-embedding-3-small
For more details, refer to the Search Documentation and Embeddings Documentation.
\nDedicated Query Thread Pool: Query execution and accelerated refreshes now run on their own dedicated thread pool, separate from the HTTP server. This prevents heavy query workloads from slowing down API responses, keeping health checks fast and avoiding unnecessary Kubernetes pod restarts under load.
\nThis feature was opt-in in previous releases and is now enabled by default. To disable it and revert to the previous behavior, add the following spicepod.yaml configuration:
\nruntime:
params:
dedicated_thread_pool: none
For more details, refer to the Runtime Configuration Documentation.
\nStale-While-Revalidate Cache Control: Query results now support \"stale-while-revalidate\" cache control, allowing stale cached data to be served immediately while asynchronously refreshing the cache entry in the background. This improves response times for frequently-accessed queries while maintaining data freshness. Requires cache key type to be set to \"sql (raw)\" for proper operation.
\nOptimized Prepared Statements: Prepared statement handling has been optimized for better performance with parameterized queries, reducing planning overhead and improving execution time for repeated queries.
\nLarge RecordBatch Chunking: Large Arrow RecordBatch objects are now automatically chunked to control memory usage during query execution, preventing memory exhaustion for queries returning large result sets.
Zstd Compression Encoding: Query result caching now supports optional Zstandard (zstd) compression encoding to reduce memory usage for cached query results. This is particularly beneficial for large result sets, reducing cache memory footprint while maintaining fast decompression times. Encoding can be configured via the encoding parameter with options none (default) or zstd.
Example configuration:
\nruntime:
caching:
sql_results:
enabled: true
max_size: 128MiB
item_ttl: 1m
encoding: zstd # Enable zstd compression
HTTP Cache-Control Support: The query result cache now supports the stale-while-revalidate Cache-Control directive, enabling faster response times by serving stale cached results immediately while asynchronously refreshing the cache in the background. This feature is particularly useful for applications that can tolerate slightly stale data in exchange for improved performance.
Example configuration:
\nruntime:
caching:
sql_results:
enabled: true
max_size: 128MiB
item_ttl: 1m
stale_while_revalidate_ttl: 1m # serve stale items for up to 1 minute after `item_ttl` expires
How it works:
\nWhen a cache entry is stale but within the stale-while-revalidate window, Spice will:
Configuration:
\nUse the Cache-Control HTTP header with the stale-while-revalidate directive:
Cache-Control: max-age=300, stale-while-revalidate=60
This configuration caches results for 5 minutes (300 seconds), and allows serving stale results for an additional 60 seconds while refreshing in the background.
\nRequirements:
\ncache_key_type to sql or plan in results_caching configuration)Example configuration via HTTP header:
\nGET /v1/sql
Cache-Control: max-age=600, stale-while-revalidate=120
X-Cache-Key-Type: sql
This feature improves application responsiveness while ensuring data freshness through background updates.
\nFor more details, refer to the Results Caching Documentation.
\nEnhanced HTTP Client Security: HTTP client usage across the runtime has been hardened with improved TLS validation, certificate pinning for critical endpoints, and better error handling for network failures.
\nODBC Connector Improvements: Removed unwrap calls from the ODBC connector, improving error handling and reliability. Fixed secret handling and Kubernetes secret integration.
\nCLI Permissions Hardening: Tightened file permissions for the CLI and install script, ensuring secure defaults for configuration files and credentials.
\nOracle Instant Client Pinning: Oracle Instant Client downloads are now pinned to specific SHAs, ensuring reproducible builds and preventing supply chain attacks.
\nImproved Credential Retry Logic: AWS SDK credential initialization has been significantly improved with more robust retry logic and better error handling. The system now automatically retries transient credential resolution failures using Fibonacci backoff, allowing Spice to tolerate extended AWS outages (up to ~48 hours) without manual intervention.
\nKey features:
\nThe improvements ensure more reliable AWS service integration, particularly in environments with intermittent network connectivity or during AWS service degradations.
\nDataFusion Log Emission: The Spice runtime now emits DataFusion internal logs, providing deeper visibility into query planning and execution for debugging and performance analysis.
\nAI Completions Tracing: Fixed tracing so that ai_completions operations are correctly parented under sql_query traces, improving observability for AI-powered queries.
Version-Controlled Data Access: The new Git Data Connector (Alpha) enables querying datasets stored in Git repositories. This connector is ideal for use cases involving configuration files, documentation, or any data tracked in version control.
\nExample Spicepod.yml configuration:
\ndatasets:
- from: git:https://github.com/myorg/myrepo
name: git_metrics
params:
file_format: csv
For more details, refer to the Git Data Connector Documentation.
\nThe Spice Java SDK has been upgraded with support for configurable Arrow memory limit: spice-java v0.4.0
\nSpiceClient client = SpiceClient.builder()
.withArrowMemoryLimitMB(1024) // 1GB limit
.build();
For more details, refer to the Java SDK Documentation.
\nInstall Specific Versions: The spice install command now supports installing specific versions of the Spice runtime and CLI. This enables easy version management, downgrading, or installation of specific releases for testing or compatibility requirements.
Usage:
\n# Install a specific version
spice install v1.8.3
# Install a specific version with AI flavor
spice install v1.8.3 ai
# Install latest version (existing behavior)
spice install
spice install ai
Note: Homebrew installations require manual version management via brew install spiceai/spiceai/spice@<version>.
Persistent Query History: The Spice CLI REPL (SQL, search, and chat interfaces) now persists command history to ~/.spice/query_history.txt, making your query history available across sessions. The history file is automatically created if it doesn't exist, with graceful fallback if the home directory cannot be determined.
New REPL Commands:
\n.clear - Clear the screen using ANSI escape codes for a clean workspace.clear history - Clear and persist the query history, removing all stored commandsTab Completion: Tab completion now includes suggestions based on your command history, making it faster to re-run or modify previous queries.
\nExample usage:
\nsql> SELECT * FROM my_table;
sql> .clear # Clears the screen
sql> .clear history # Clears command history
sql> # Use arrow keys or tab to access previous commands
For more details, refer to the CLI Documentation.
\nspicepod.yaml files, providing actionable feedback for misconfiguration.ListingTable partitions are pruned correctly when filters are not used.regexp_match function for DuckDB datasets.No breaking changes.
\nNew HTTP Data Connector Recipe: New recipe demonstrating how to query REST APIs and HTTP(s) endpoints. See HTTP Connector Recipe for details.
\nThe Spice Cookbook includes 82 recipes to help you get started with Spice quickly and easily.
\nTo upgrade to v1.9.0, use one of the following methods:
\nCLI:
\nspice upgrade
Homebrew:
\nbrew upgrade spiceai/spiceai/spice
Docker:
\nPull the spiceai/spiceai:1.9.0 image:
docker pull spiceai/spiceai:1.9.0
For available tags, see DockerHub.
\nHelm:
\nhelm repo update
helm upgrade spiceai spiceai/spiceai
AWS Marketplace:
\nπ Spice is now available in the AWS Marketplace!
\nruntime-secrets crate by @phillipleblanc in #7474runtime-parameters crate by @phillipleblanc in #7475spice sql by @lukekim in #7479runtime-request-context crate by @Jeadie in #7459spill_compression to runtime config by @krinart in #7505RequestTimeoutException to S3 client by @Jeadie in #7514sha=<> to snapshot logging by @phillipleblanc in #7521spice chat play nice with Unix pipes by @Jeadie in #7525runtime-async crate with managed Tokio runtime by @phillipleblanc in #7575runtime.metrics config by @krinart in #7583ObjectMeta filter pushdown for ObjectStoreTextTable by @Jeadie in #7572TableProvider from CandidateGeneration::search. by @Jeadie in #7559Build Docker Image from PR checks by @phillipleblanc in #7621duckdb_partition_mode to partition_mode param by @sgrebnov in #7622query_active_count metric by @krinart in #7589min_sql_duration filter support by @lukekim in #7698object_store IO tasks on the original Tokio runtime by @phillipleblanc in #7689dedicated_thread_pool by @phillipleblanc in #7705connection_pool_size param by @sgrebnov in #7716AcceleratedTable by @Jeadie in #7724on_refresh_recompute_statistics param + ANALYZE for table-based partitioning by @sgrebnov in #7719AWS_SESSION_TOKEN parameter by @mach-kernel in #7752Scan request by @krinart in #7829supports_limit_pushdown for SchemaCastScanExec by @sgrebnov in #7893dataset_acceleration_last_refresh_time_ms unit to milliseconds in description by @ewgenius in #7901AWS_EC2_METADATA_DISABLED flag by @sgrebnov in #7952JoinSelection is applied by @peasee in #7828join_key from TableProvider::constraints and implement SearchQueryProvider::constraints. by @Jeadie in #7959digest_many UDF for ColumnarValue::Array. by @Jeadie in #7960.clear and .clear history commands by @lukekim in #7970stale-while-revalidate by @lukekim in #7963cluster a default feature by @lukekim in #7994RequestContext for cluster queries by @Jeadie in #8007generate_changelog script by @krinart in #8028CachedQueryVector to avoid recomputing embedding vector for spilling/partitioned vector indexes. by @Jeadie in #8059verify_results dispatch configuration by @sgrebnov in #8106delta_kernel::listed_log_files warnings by @phillipleblanc in #8158Announcing the release of Spice v1.9.0-rc.4! πΆ
\nThis release candidate brings DuckDB v1.4.2, Cayenne partitioning improvements, and comprehensive security hardening across the CLI, data connectors, runtime, and MCP. v1.9.0-rc.4 also includes MySQL and PostgreSQL connector improvements with fixed nullability inferences and full-text search support, DynamoDB consistency improvements, HTTP connector validation and UX enhancements, and numerous reliability and performance optimizations. Significant improvements were also made to test and automation infrastructure to ensure high quality releases.
\nv1.9.0 introduces Spice Cayenne, a new high-performance data accelerator built on the Vortex columnar format that delivers better than DuckDB performance without single-file scaling limitations, and a preview of Multi-Node Distributed Query based on Apache Ballista. v1.9.0 also upgrades to DataFusion v50 for even higher query performance, expands search capabilities with full-text search on views and multi-column embeddings, and delivers many additional features and improvements.
\nIntroducing Cayenne: SQL as an Acceleration Format: A new high-performance Data Accelerator that simplifies multi-file data acceleration by using an embedded database (SQLite) for metadata while storing data in the Vortex columnar format, a Linux Foundation project. Cayenne delivers query and ingestion performance better than DuckDB's file-based acceleration without DuckDB's memory overhead and the scaling challenges of single DuckDB files.
\nCayenne uses SQLite to manage acceleration metadata (schemas, snapshots, statistics, file tracking) through simple SQL transactions, while storing data in Vortex's compressed columnar format. This architecture provides:
\nKey Features:
\nExample Spicepod.yml configuration:
\ndatasets:
- from: s3:my_table
name: accelerated_data_30d
acceleration:
enabled: true
engine: cayenne
mode: file
refresh_mode: append
retention_sql: DELETE FROM accelerated_data WHERE created_at < NOW() - INTERVAL '30 days'
Note, the Cayenne Data Accelerator is in Beta with limitations.
\nFor more details, refer to the Cayenne Documentation, the Vortex project, and the DuckLake announcement that partly inspired this design.
\nApache Ballista Integration: Spice now supports distributed query execution based on Apache Ballista, enabling distributed queries across multiple executor nodes for improved performance on large datasets. This feature is in preview in v1.9.0-rc.3.
\nArchitecture:
\nA distributed Spice cluster consists of:
\nGetting Started:
\nStart a scheduler instance using an existing Spicepod. The scheduler is the only spiced instance that needs to be configured:
\n# Start scheduler (note the flight bind address override if you want it reachable outside localhost)
spiced --cluster-mode scheduler --flight 0.0.0.0:50051
Start one or more executors configured with the scheduler's flight URI:
\n# Start executor (automatically selects a free port if 50051 is taken)
spiced --cluster-mode executor --scheduler-url spiced://localhost:50051
Query Execution:
\nQueries run through the scheduler will now show a distributed_plan in EXPLAIN output, demonstrating how the query is distributed across executor nodes:
EXPLAIN SELECT count(id) FROM my_dataset;
Current Limitations:
\nSpice.ai is built on the Apache DataFusion query engine. The v50 release brings significant performance improvements and enhanced reliability:
\nPerformance Improvements π:
\nDynamic Filter Pushdown: Enhanced dynamic filter pushdown for custom ExecutionPlans, ensuring filters propagate correctly through all physical operators for improved query performance.
Partition Pruning: Expanded partition pruning support ensures that unnecessary partitions are skipped when filters are not used, reducing data scanning overhead and improving query execution times.
\nApache Spark Compatible Functions: Added support for Spark-compatible functions including array, bit_get/bit_count, bitmap_count, crc32/sha1, date_add/date_sub, if, last_day, like/ilike, luhn_check, mod/pmod, next_day, parse_url, rint, and width_bucket.
Bug Fixes & Reliability: Resolved issues with partition name validation and empty execution plans when vector index lists are empty. Fixed timestamp support for partition expressions, enabling better partitioning for time-series data.
\nSee the Apache DataFusion 50.0.3 Release for more details.
\nDuckDB v1.4.2: DuckDB has been upgraded to v1.4.2, which includes several performance optimizations.
\nComposite ART Index Support: DuckDB in Spice now supports composite (multi-column) Adaptive Radix Tree (ART) indexes for accelerated table scans. When queries filter on multiple columns fully covered by a composite index, the optimizer automatically uses index scans instead of full table scans, delivering significant performance improvements for selective queries.
\nExample configuration:
\ndatasets:
- from: file://data.parquet
name: sales
acceleration:
enabled: true
engine: duckdb
indexes:
'(region, product_id)': enabled
Performance example with composite index on 7.5M rows:
\nSELECT * FROM sales WHERE region = 'US' AND product_id = 12345;
-- Without index: 0.282s
-- With composite index (region, product_id): 0.037s
-- Performance improvement: 7.6x faster with composite index
DuckDB Intermediate Materialization: Queries with indexes now use intermediate materialization (WITH ... AS MATERIALIZED) to leverage faster index scans. Currently supported for non-federated queries (query_federation: disabled) against a single table with indexes only. When predicates cover more columns than the index, the optimizer rewrites queries to first materialize index-filtered results, then apply remaining predicates. This optimization can deliver significant performance improvements for selective queries.
Example configuration:
\ndatasets:
- from: file://sales_data.parquet
name: sales
acceleration:
enabled: true
engine: duckdb
mode: file
params:
query_federation: disabled # Required currently for intermediate materialization
indexes:
'(region, product_id)': enabled
Performance example:
\n-- Query with indexed columns (region, product_id) plus additional filter (amount)
SELECT * FROM sales
WHERE region = 'US' AND product_id = 12345 AND amount > 1000;
-- Optimized execution time: 0.031s (with intermediate materialization)
-- Standard execution time: 0.108s (without optimization)
-- Performance improvement: ~3.5x faster
The optimizer automatically rewrites the query to:
\nWITH _intermediate_materialize AS MATERIALIZED (
SELECT * FROM sales WHERE region = 'US' AND product_id = 12345
)
SELECT * FROM _intermediate_materialize WHERE amount > 1000;
Parquet Buffering for Partitioned Writes: DuckDB partitioned writes in table mode now support Parquet buffering, reducing memory usage and improving write performance for large datasets.
\nRetention SQL on Refresh Commit: DuckDB accelerations now support running retention SQL on refresh commit, enabling automatic data cleanup and lifecycle management during refresh operations.
\nUTC Timezone for DuckDB: DuckDB now uses UTC as the default timezone, ensuring consistent behavior for time-based queries across different environments.
\nExample Spicepod.yml configuration:
\ndatasets:
- from: s3://my_bucket/large_table/
name: partitioned_data
acceleration:
enabled: true
engine: duckdb
mode: file
retention:
sql: DELETE FROM partitioned_data WHERE event_time < NOW() - INTERVAL '7 days'
Querying endpoints as tables: The HTTP/HTTPS Data Connectors now supports querying HTTP endpoints directly as tables in SQL queries with dynamic filters. This feature transforms REST APIs into queryable data sources, making it easy to integrate external service data.
\nQuery HTTP endpoint that returns structured data (JSON, CSV, etc.) as if it were a database table
\nConfigurable retry logic, timeouts, and POST request support for more complex API interactions
\nExample Spicepod.yml configuration:
\ndatasets:
- from: https://api.tvmaze.com
name: tvmaze
params:
file_format: json
max_retries: 3
client_timeout: 10s
Example SQL query:
\nSELECT request_path, request_query, content
FROM tvmaze
WHERE request_path = '/search/people' and request_query = 'q=michael'
LIMIT 10;
If a request_body is supplied it will be posted to the endpoint:
Example SQL query:
\nSELECT request_path, request_query, content
FROM tvmaze
WHERE request_path = '/search/people' and request_query = 'q=michael' and request_body = '{\"name\": \"michael\"}'
LIMIT 10;
HTTP endpoints can be accelerated using refresh_sql:
datasets:
- from: https://api.tvmaze.com
name: tvmaze
acceleration:
enabled: true
refresh_mode: full
refresh_sql: |
SELECT request_path, request_query, content
FROM tvmaze
WHERE request_path = '/search/people'
AND request_query IN ('q=michael', 'q=luke')
Improved Query Performance: The DynamoDB Data Connector now includes improved filter handling for edge cases, parallel scan support for faster data ingestion, and better error handling for misconfigured queries. These improvements enable more reliable and performant access to DynamoDB data.
\nExample Spicepod.yml configuration:
\ndatasets:
- from: dynamodb:my_table
name: ddb_data
params:
scan_segments: 10 # Default `auto` which calculates optimal segments based on number of rows
Atomic Range Reads for Versioned Files: Spice now supports S3 Versioning for all connectors using object-store (S3, Delta Lake, etc.), ensuring range reads over versioned files are atomically correct. When S3 versioning is enabled, Spice automatically tracks version IDs during file discovery and uses them for all subsequent range reads, preventing inconsistencies from concurrent file modifications.
\nCurrent limitations:
\nFull-Text Search on Views: Full-text search indexes are now supported on views, enabling advanced search scenarios over pre-aggregated or transformed data. This extends the power of Spice's search capabilities beyond base datasets.
\nMulti-Column Embeddings on Views: Views now support embedding columns, enabling vector search and semantic retrieval on view data. This is useful for search over aggregated or joined datasets.
\nVector Engines on Views: Vector search engines are now available for views, enabling similarity search over complex queries and transformations.
\nExample Spicepod.yml configuration:
\nviews:
- name: aggregated_reviews
sql: SELECT review_id, review_text FROM reviews WHERE rating > 4
embeddings:
- column: review_text
model: openai:text-embedding-3-small
Dedicated Query Thread Pool: Query execution and accelerated refreshes now run on their own dedicated thread pool, separate from the HTTP server. This prevents heavy query workloads from slowing down API responses, keeping health checks fast and avoiding unnecessary Kubernetes pod restarts under load.
\nThis feature was opt-in in previous releases and is now enabled by default. To disable it and revert to the previous behavior, add the following spicepod.yaml configuration:
\nruntime:
params:
dedicated_thread_pool: none
Stale-While-Revalidate Cache Control: Query results now support \"stale-while-revalidate\" cache control, allowing stale cached data to be served immediately while asynchronously refreshing the cache entry in the background. This improves response times for frequently-accessed queries while maintaining data freshness. Requires cache key type to be set to \"sql (raw)\" for proper operation.
\nOptimized Prepared Statements: Prepared statement handling has been optimized for better performance with parameterized queries, reducing planning overhead and improving execution time for repeated queries.
\nLarge RecordBatch Chunking: Large Arrow RecordBatch objects are now automatically chunked to control memory usage during query execution, preventing memory exhaustion for queries returning large result sets.
HTTP Cache-Control Support: The query result cache now supports the stale-while-revalidate Cache-Control directive, enabling faster response times by serving stale cached results immediately while asynchronously refreshing the cache in the background. This feature is particularly useful for applications that can tolerate slightly stale data in exchange for improved performance.
How it works:
\nWhen a cache entry is stale but within the stale-while-revalidate window, Spice will:
Configuration:
\nUse the Cache-Control HTTP header with the stale-while-revalidate directive:
Cache-Control: max-age=300, stale-while-revalidate=60
This configuration caches results for 5 minutes (300 seconds), and allows serving stale results for an additional 60 seconds while refreshing in the background.
\nRequirements:
\ncache_key_type to sql or plan in results_caching configuration)Example configuration via HTTP header:
\nGET /v1/sql
Cache-Control: max-age=600, stale-while-revalidate=120
X-Cache-Key-Type: sql
This feature improves application responsiveness while ensuring data freshness through background updates.
\nEnhanced HTTP Client Security: HTTP client usage across the runtime has been hardened with improved TLS validation, certificate pinning for critical endpoints, and better error handling for network failures.
\nODBC Connector Improvements: Removed unwrap calls from the ODBC connector, improving error handling and reliability. Fixed secret handling and Kubernetes secret integration.
\nCLI Permissions Hardening: Tightened file permissions for the CLI and install script, ensuring secure defaults for configuration files and credentials.
\nOracle Instant Client Pinning: Oracle Instant Client downloads are now pinned to specific SHAs, ensuring reproducible builds and preventing supply chain attacks.
\nImproved Credential Retry Logic: AWS SDK credential initialization has been significantly improved with more robust retry logic and better error handling. The system now automatically retries transient credential resolution failures using Fibonacci backoff, allowing Spice to tolerate extended AWS outages (up to ~48 hours) without manual intervention.
\nKey features:
\nThe improvements ensure more reliable AWS service integration, particularly in environments with intermittent network connectivity or during AWS service degradations.
\nDataFusion Log Emission: The Spice runtime now emits DataFusion internal logs, providing deeper visibility into query planning and execution for debugging and performance analysis.
\nAI Completions Tracing: Fixed tracing so that ai_completions operations are correctly parented under sql_query traces, improving observability for AI-powered queries.
Version-Controlled Data Access: The new Git Data Connector (Alpha) enables querying datasets stored in Git repositories. This connector is ideal for use cases involving configuration files, documentation, or any data tracked in version control.
\nExample Spicepod.yml configuration:
\ndatasets:
- from: git:https://github.com/myorg/myrepo
name: git_metrics
params:
file_format: csv
For more details, refer to the Git Data Connector Documentation.
\nThe Spice Java SDK have been upgraded with support configurable Arrow memory limit: spice-java v0.4.0
\nSpiceClient client = SpiceClient.builder()
.withArrowMemoryLimitMB(1024) // 1GB limit
.build();
Install Specific Versions: The spice install command now supports installing specific versions of the Spice runtime and CLI. This enables easy version management, downgrading, or installation of specific releases for testing or compatibility requirements.
Usage:
\n# Install a specific version
spice install v1.8.3
# Install a specific version with AI flavor
spice install v1.8.3 ai
# Install latest version (existing behavior)
spice install
spice install ai
Note: Homebrew installations require manual version management via brew install spiceai/spiceai/spice@<version>.
Persistent Query History: The Spice CLI REPL (SQL, search, and chat interfaces) now persists command history to ~/.spice/query_history.txt, making your query history available across sessions. The history file is automatically created if it doesn't exist, with graceful fallback if the home directory cannot be determined.
New REPL Commands:
\n.clear - Clear the screen using ANSI escape codes for a clean workspace.clear history - Clear and persist the query history, removing all stored commandsTab Completion: Tab completion now includes suggestions based on your command history, making it faster to re-run or modify previous queries.
\nExample usage:
\nsql> SELECT * FROM my_table;
sql> .clear # Clears the screen
sql> .clear history # Clears command history
sql> # Use arrow keys or tab to access previous commands
spicepod.yaml files, providing actionable feedback for misconfiguration.ListingTable partitions are pruned correctly when filters are not used.regexp_match function for DuckDB datasets.No breaking changes.
\nNew HTTP Data Connector Recipe: New recipe demonstrating how to query REST APIs and HTTP(s) endpoints. See HTTP Connector Recipe for details.
\nThe Spice Cookbook includes 82 recipes to help you get started with Spice quickly and easily.
\nTo upgrade to v1.9.0-rc.4, use one of the following methods:
\nCLI:
\nspice upgrade
Homebrew:
\nbrew upgrade spiceai/spiceai/spice
Docker:
\nPull the spiceai/spiceai:1.9.0-rc.4 image:
docker pull spiceai/spiceai:1.9.0-rc.4
For available tags, see DockerHub.
\nHelm:
\nhelm repo update
helm upgrade spiceai spiceai/spiceai
AWS Marketplace:
\nπ Spice is now available in the AWS Marketplace!
\ngenerate_changelog script by @krinart in #8028CachedQueryVector to avoid recomputing embedding vector for spilling/partitioned vector indexes. by @Jeadie in #8059verify_results dispatch configuration by @sgrebnov in #8106delta_kernel::listed_log_files warnings by @phillipleblanc in #8158Announcing the release of Spice v1.9.0-rc.2! πΆ
\nThis is the second release candidate for v1.9.0, which introduces Spice Cayenne, a new high-performance data accelerator built on the Vortex columnar format that delivers better than DuckDB performance without single-file scaling limitations and a preview of Multi-Node Distributed Query based on Apache Ballista. v1.9.0-rc.2 also upgrades to DataFusion v50 and DuckDB v1.4.1 for even higher query performance, expands search capabilities with full-text search on views and multi-column embeddings, includes significant DynamoDB and DuckDB accelerator improvements, expands the HTTP data connector to support endpoints as tables, and delivers many security and reliability improvements.
\nIntroducing Cayenne: SQL as an Acceleration Format: A new high-performance Data Accelerator that simplifies multi-file data acceleration by using an embedded database (SQLite) for metadata while storing data in the Vortex columnar format, a Linux Foundation project. Cayenne delivers query and ingestion performance better than DuckDB's file-based acceleration without DuckDB's memory overhead and the scaling challenges of single DuckDB files.
\nCayenne uses SQLite to manage acceleration metadata (schemas, snapshots, statistics, file tracking) through simple SQL transactions, while storing data in Vortex's compressed columnar format. This architecture provides:
\nKey Features:
\nExample Spicepod.yml configuration:
\ndatasets:
- from: s3:my_table
name: accelerated_data_30d
acceleration:
enabled: true
engine: cayenne
mode: file
refresh_mode: append
retention_sql: DELETE FROM accelerated_data WHERE created_at < NOW() - INTERVAL '30 days'
Note, the Cayenne Data Accelerator is in Beta with limitations.
\nFor more details, refer to the Cayenne Documentation, the Vortex project, and the DuckLake announcement that partly inspired this design.
\nApache Ballista Integration: Spice now supports distributed query execution based on Apache Ballista, enabling distributed queries across multiple executor nodes for improved performance on large datasets. This feature is in preview in v1.9.0-rc.2.
\nArchitecture:
\nA distributed Spice cluster consists of:
\nGetting Started:
\nStart a scheduler instance using an existing Spicepod. The scheduler is the only spiced instance that needs to be configured:
\n# Start scheduler (note the flight bind address override if you want it reachable outside localhost)
spiced --cluster-mode scheduler --flight 0.0.0.0:50051
Start one or more executors configured with the scheduler's flight URI:
\n# Start executor (automatically selects a free port if 50051 is taken)
spiced --cluster-mode executor --scheduler-url spiced://localhost:50051
Query Execution:
\nQueries run through the scheduler will now show a distributed_plan in EXPLAIN output, demonstrating how the query is distributed across executor nodes:
EXPLAIN SELECT count(id) FROM my_dataset;
Current Limitations:
\nSpice.ai is built on the Apache DataFusion query engine. The v50 release brings significant performance improvements and enhanced reliability:
\nPerformance Improvements π:
\nDynamic Filter Pushdown: Enhanced dynamic filter pushdown for custom ExecutionPlans, ensuring filters propagate correctly through all physical operators for improved query performance.
Partition Pruning: Expanded partition pruning support ensures that unnecessary partitions are skipped when filters are not used, reducing data scanning overhead and improving query execution times.
\nApache Spark Compatible Functions: Added support for Spark-compatible functions including array, bit_get/bit_count, bitmap_count, crc32/sha1, date_add/date_sub, if, last_day, like/ilike, luhn_check, mod/pmod, next_day, parse_url, rint, and width_bucket.
Bug Fixes & Reliability: Resolved issues with partition name validation and empty execution plans when vector index lists are empty. Fixed timestamp support for partition expressions, enabling better partitioning for time-series data.
\nSee the Apache DataFusion 50.0.0 Release for more details.
\nDuckDB v1.4.1: DuckDB has been upgraded to v1.4.1, which includes several performance optimizations.
\nComposite ART Index Support: DuckDB in Spice now supports composite (multi-column) Adaptive Radix Tree (ART) indexes for accelerated table scans. When queries filter on multiple columns fully covered by a composite index, the optimizer automatically uses index scans instead of full table scans, delivering significant performance improvements for selective queries.
\nExample configuration:
\ndatasets:
- from: file://data.parquet
name: sales
acceleration:
enabled: true
engine: duckdb
indexes:
'(region, product_id)': enabled
Performance example with composite index on 7.5M rows:
\nSELECT * FROM sales WHERE region = 'US' AND product_id = 12345;
-- Without index: 0.282s
-- With composite index (region, product_id): 0.037s
-- Performance improvement: 7.6x faster with composite index
DuckDB Intermediate Materialization: Queries with indexes now use intermediate materialization (WITH ... AS MATERIALIZED) to leverage faster index scans. Currently supported for non-federated queries (query_federation: disabled) against a single table with indexes only. When predicates cover more columns than the index, the optimizer rewrites queries to first materialize index-filtered results, then apply remaining predicates. This optimization can deliver significant performance improvements for selective queries.
Example configuration:
\ndatasets:
- from: file://sales_data.parquet
name: sales
acceleration:
enabled: true
engine: duckdb
mode: file
params:
query_federation: disabled # Required currently for intermediate materialization
indexes:
'(region, product_id)': enabled
Performance example:
\n-- Query with indexed columns (region, product_id) plus additional filter (amount)
SELECT * FROM sales
WHERE region = 'US' AND product_id = 12345 AND amount > 1000;
-- Optimized execution time: 0.031s (with intermediate materialization)
-- Standard execution time: 0.108s (without optimization)
-- Performance improvement: ~3.5x faster
The optimizer automatically rewrites the query to:
\nWITH _intermediate_materialize AS MATERIALIZED (
SELECT * FROM sales WHERE region = 'US' AND product_id = 12345
)
SELECT * FROM _intermediate_materialize WHERE amount > 1000;
Parquet Buffering for Partitioned Writes: DuckDB partitioned writes in table mode now support Parquet buffering, reducing memory usage and improving write performance for large datasets.
\nRetention SQL on Refresh Commit: DuckDB accelerations now support running retention SQL on refresh commit, enabling automatic data cleanup and lifecycle management during refresh operations.
\nUTC Timezone for DuckDB: DuckDB now uses UTC as the default timezone, ensuring consistent behavior for time-based queries across different environments.
\nExample Spicepod.yml configuration:
\ndatasets:
- from: s3://my_bucket/large_table/
name: partitioned_data
acceleration:
enabled: true
engine: duckdb
mode: file
retention:
sql: DELETE FROM partitioned_data WHERE event_time < NOW() - INTERVAL '7 days'
Querying endpoints as tables: The HTTP/HTTPS Data Connectors now supports querying HTTP endpoints directly as tables in SQL queries with dynamic filters. This feature transforms REST APIs into queryable data sources, making it easy to integrate external service data.
\nQuery HTTP endpoint that returns structured data (JSON, CSV, etc.) as if it were a database table
\nConfigurable retry logic, timeouts, and POST request support for more complex API interactions
\nExample Spicepod.yml configuration:
\ndatasets:
- from: https://api.tvmaze.com
name: tvmaze
params:
file_format: json
max_retries: 3
client_timeout: 10s
Example SQL query:
\nSELECT request_path, request_query, content
FROM tvmaze
WHERE request_path = '/search/people' and request_query = 'q=michael'
LIMIT 10;
If a request_body is supplied it will be posted to the endpoint:
Example SQL query:
\nSELECT request_path, request_query, content
FROM tvmaze
WHERE request_path = '/search/people' and request_query = 'q=michael' and request_body = '{\"name\": \"michael\"}'
LIMIT 10;
HTTP endpoints can be accelerated using refresh_sql:
datasets:
- from: https://api.tvmaze.com
name: tvmaze
acceleration:
enabled: true
refresh_mode: full
refresh_sql: |
SELECT request_path, request_query, content
FROM tvmaze
request_path = '/search/people'
AND request_query IN ('q=michael', 'q=luke')
Improved Query Performance: The DynamoDB Data Connector now includes improved filter handling for edge cases, parallel scan support for faster data ingestion, and better error handling for misconfigured queries. These improvements enable more reliable and performant access to DynamoDB data.
\nExample Spicepod.yml configuration:
\ndatasets:
- from: dynamodb:my_table
name: ddb_data
params:
scan_segments: 10 # Default `auto` which calculates optimal segments based on number of rows
Atomic Range Reads for Versioned Files: Spice now supports S3 Versioning for all connectors using object-store (S3, Delta Lake, etc.), ensuring range reads over versioned files are atomically correct. When S3 versioning is enabled, Spice automatically tracks version IDs during file discovery and uses them for all subsequent range reads, preventing inconsistencies from concurrent file modifications.
\nCurrent limitations:
\nFull-Text Search on Views: Full-text search indexes are now supported on views, enabling advanced search scenarios over pre-aggregated or transformed data. This extends the power of Spice's search capabilities beyond base datasets.
\nMulti-Column Embeddings on Views: Views now support embedding columns, enabling vector search and semantic retrieval on view data. This is useful for search over aggregated or joined datasets.
\nVector Engines on Views: Vector search engines are now available for views, enabling similarity search over complex queries and transformations.
\nExample Spicepod.yml configuration:
\nviews:
- name: aggregated_reviews
sql: SELECT review_id, review_text FROM reviews WHERE rating > 4
embeddings:
- column: review_text
model: openai:text-embedding-3-small
Dedicated Query Thread Pool: Query execution and accelerated refreshes now run on their own dedicated thread pool, separate from the HTTP server. This prevents heavy query workloads from slowing down API responses, keeping health checks fast and avoiding unnecessary Kubernetes pod restarts under load.
\nThis feature was opt-in in previous releases and is now enabled by default in v1.9.0-rc.2. To disable it and revert to the previous behavior, add the following spicepod.yaml configuration:
\nruntime:
params:
dedicated_thread_pool: none
Stale-While-Revalidate Cache Control: Query results now support \"stale-while-revalidate\" cache control, allowing stale cached data to be served immediately while asynchronously refreshing the cache entry in the background. This improves response times for frequently-accessed queries while maintaining data freshness. Requires cache key type to be set to \"sql (raw)\" for proper operation.
\nOptimized Prepared Statements: Prepared statement handling has been optimized for better performance with parameterized queries, reducing planning overhead and improving execution time for repeated queries.
\nLarge RecordBatch Chunking: Large Arrow RecordBatch objects are now automatically chunked to control memory usage during query execution, preventing memory exhaustion for queries returning large result sets.
HTTP Cache-Control Support: The query result cache now supports the stale-while-revalidate Cache-Control directive, enabling faster response times by serving stale cached results immediately while asynchronously refreshing the cache in the background. This feature is particularly useful for applications that can tolerate slightly stale data in exchange for improved performance.
How it works:
\nWhen a cache entry is stale but within the stale-while-revalidate window, Spice will:
Configuration:
\nUse the Cache-Control HTTP header with the stale-while-revalidate directive:
Cache-Control: max-age=300, stale-while-revalidate=60
This configuration caches results for 5 minutes (300 seconds), and allows serving stale results for an additional 60 seconds while refreshing in the background.
\nRequirements:
\ncache_key_type to sql or plan in results_caching configuration)Example configuration via HTTP header:
\nGET /v1/sql
Cache-Control: max-age=600, stale-while-revalidate=120
X-Cache-Key-Type: sql
This feature improves application responsiveness while ensuring data freshness through background updates.
\nEnhanced HTTP Client Security: HTTP client usage across the runtime has been hardened with improved TLS validation, certificate pinning for critical endpoints, and better error handling for network failures.
\nODBC Connector Improvements: Removed unwrap calls from the ODBC connector, improving error handling and reliability. Fixed secret handling and Kubernetes secret integration.
\nCLI Permissions Hardening: Tightened file permissions for the CLI and install script, ensuring secure defaults for configuration files and credentials.
\nOracle Instant Client Pinning: Oracle Instant Client downloads are now pinned to specific SHAs, ensuring reproducible builds and preventing supply chain attacks.
\nImproved Credential Retry Logic: AWS SDK credential initialization has been significantly improved with more robust retry logic and better error handling. The system now automatically retries transient credential resolution failures using Fibonacci backoff, allowing Spice to tolerate extended AWS outages (up to ~48 hours) without manual intervention.
\nKey features:
\nThe improvements ensure more reliable AWS service integration, particularly in environments with intermittent network connectivity or during AWS service degradations.
\nDataFusion Log Emission: The Spice runtime now emits DataFusion internal logs, providing deeper visibility into query planning and execution for debugging and performance analysis.
\nAI Completions Tracing: Fixed tracing so that ai_completions operations are correctly parented under sql_query traces, improving observability for AI-powered queries.
Version-Controlled Data Access: The new Git Data Connector (Alpha) enables querying datasets stored in Git repositories. This connector is ideal for use cases involving configuration files, documentation, or any data tracked in version control.
\nExample Spicepod.yml configuration:
\ndatasets:
- from: git:https://github.com/myorg/myrepo
name: git_metrics
params:
file_format: csv
For more details, refer to the Git Data Connector Documentation.
\nThe Spice Java SDK have been upgraded with support configurable Arrow memory limit: spice-java v0.4.0
\nSpiceClient client = SpiceClient.builder()
.withArrowMemoryLimitMB(1024) // 1GB limit
.build();
Install Specific Versions: The spice install command now supports installing specific versions of the Spice runtime and CLI. This enables easy version management, downgrading, or installation of specific releases for testing or compatibility requirements.
Usage:
\n# Install a specific version
spice install v1.8.3
# Install a specific version with AI flavor
spice install v1.8.3 ai
# Install latest version (existing behavior)
spice install
spice install ai
Note: Homebrew installations require manual version management via brew install spiceai/spiceai/spice@<version>.
Persistent Query History: The Spice CLI REPL (SQL, search, and chat interfaces) now persists command history to ~/.spice/query_history.txt, making your query history available across sessions. The history file is automatically created if it doesn't exist, with graceful fallback if the home directory cannot be determined.
New REPL Commands:
\n.clear - Clear the screen using ANSI escape codes for a clean workspace.clear history - Clear and persist the query history, removing all stored commandsTab Completion: Tab completion now includes suggestions based on your command history, making it faster to re-run or modify previous queries.
\nExample usage:
\nsql> SELECT * FROM my_table;
sql> .clear # Clears the screen
sql> .clear history # Clears command history
sql> # Use arrow keys or tab to access previous commands
spicepod.yaml files, providing actionable feedback for misconfiguration.ListingTable partitions are pruned correctly when filters are not used.regexp_match function for DuckDB datasets.No breaking changes.
\nNew HTTP Data Connector Recipe: New recipe demonstrating how to query REST APIs and HTTP(s) endpoints. See HTTP Connector Recipe for details.
\nThe Spice Cookbook includes 82 recipes to help you get started with Spice quickly and easily.
\nTo upgrade to v1.9.0-rc.2, use one of the following methods:
\nCLI:
\nspice upgrade
Homebrew:
\nbrew upgrade spiceai/spiceai/spice
Docker:
\nPull the spiceai/spiceai:1.9.0-rc.2 image:
docker pull spiceai/spiceai:1.9.0-rc.2
For available tags, see DockerHub.
\nHelm:
\nhelm repo update
helm upgrade spiceai spiceai/spiceai
AWS Marketplace:
\nπ Spice is now available in the AWS Marketplace!
\nruntime-secrets crate by @phillipleblanc in #7474runtime-parameters crate by @phillipleblanc in #7475Scan request by @krinart in #7829supports_limit_pushdown for SchemaCastScanExec by @sgrebnov in #7893dataset_acceleration_last_refresh_time_ms unit to milliseconds in description by @ewgenius in #7901AWS_EC2_METADATA_DISABLED flag by @sgrebnov in #7952JoinSelection is applied by @peasee in #7828join_key from TableProvider::constraints and implement SearchQueryProvider::constraints. by @Jeadie in #7959digest_many UDF for ColumnarValue::Array. by @Jeadie in #7960.clear and .clear history commands by @lukekim in #7970stale-while-revalidate by @lukekim in #7963cluster a default feature by @lukekim in #7994RequestContext for cluster queries by @Jeadie in #8007