32 posts tagged with "data-connector"

The v0.13.0-alpha release significantly improves federated query performance and efficiency with Query Push-Down. Query push-down allows SQL queries to be directly executed by underlying data sources, such as joining tables using the same data connector. Query push-down is supported for all SQL-based and Arrow Flight data connectors. Additionally, runtime metrics, including query duration, collected and accessed in the spice.runtime.metrics table. This release also includes a new FTP/SFTP data connector and improved CSV support for the S3 data connector.

Highlights​

• Federated Query Push-Down (#1394): All SQL and Arrow Flight data connectors support federated query push-down.

• Runtime Metrics (#1361): Runtime metric collection can be enabled using the --metrics flag and accessed by the spice.runtime.metrics table.

• FTP & SFTP data connector (#1355) (#1399): Added support for using FTP and SFTP as data sources.

• Improved CSV support (#1411) (#1414): S3/FTP/SFTP data connectors support CSV files with expanded CSV options.

Contributors​

• @Jeadie
• @digadeesh
• @ewgenius
• @gloomweaver
• @lukekim
• @phillipleblanc
• @sgrebnov
• @y-f-u

What's Changed​

• Remove milestones from Enhancement template by @lukekim in https://github.com/spiceai/spiceai/pull/1373
• Update version.txt and Cargo.toml to 0.13.0-alpha by @sgrebnov in https://github.com/spiceai/spiceai/pull/1375
• Helm chart for Spice v0.12.2-alpha by @sgrebnov in https://github.com/spiceai/spiceai/pull/1374
• Add release cargo feature to docker builds by @ewgenius in https://github.com/spiceai/spiceai/pull/1377
• FTP connector by @gloomweaver in https://github.com/spiceai/spiceai/pull/1355
• Provide ability to specify timeout for s3 data connector by @gloomweaver in https://github.com/spiceai/spiceai/pull/1378
• clickhouse-rs use tag instead of branch by @gloomweaver in https://github.com/spiceai/spiceai/pull/1313
• Store runtime metrics in spice.runtime.metrics table by @ewgenius in https://github.com/spiceai/spiceai/pull/1361
• Update bug_report.md to include the kind/bug label by @digadeesh in https://github.com/spiceai/spiceai/pull/1381
• Remove redundant [refresh] in log by @lukekim in https://github.com/spiceai/spiceai/pull/1384
• Implement federation for DuckDB Data Connector (POC) by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1380
• Update wording for spice cloud connection by @ewgenius in https://github.com/spiceai/spiceai/pull/1386
• fix dataset refreshing status by @y-f-u in https://github.com/spiceai/spiceai/pull/1387
• clickhouse friendly error by @y-f-u in https://github.com/spiceai/spiceai/pull/1388
• Initial work for NQL crate and API by @Jeadie in https://github.com/spiceai/spiceai/pull/1366
• Fully implement federation for all SqlTable-based Data Connectors by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1394
• use df logical plan to query latest timestamp when refreshing incrementally by @y-f-u in https://github.com/spiceai/spiceai/pull/1393
• Refactor datafusion.write_data to use table reference by @ewgenius in https://github.com/spiceai/spiceai/pull/1402
• Add federation to FlightTable based DataConnectors by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1401
• SFTP Data Connector by @gloomweaver in https://github.com/spiceai/spiceai/pull/1399
• Use GPT3.5 for NSQL task by @Jeadie in https://github.com/spiceai/spiceai/pull/1400
• Update ROADMAP May 16, 2024 by @lukekim in https://github.com/spiceai/spiceai/pull/1405
• Add ftp/sftp connector to readme by @gloomweaver in https://github.com/spiceai/spiceai/pull/1404
• Add FlightSQL federation provider by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1403
• Refactor runtime metrics to use localhost accelerated table by @ewgenius in https://github.com/spiceai/spiceai/pull/1395
• Use JSON response in OpenAI, text -> SQL model by @Jeadie in https://github.com/spiceai/spiceai/pull/1407
• support more common csv options by @y-f-u in https://github.com/spiceai/spiceai/pull/1411
• add a TLS error message in data connector and implement it for clickhouse by @y-f-u in https://github.com/spiceai/spiceai/pull/1413
• Add CSV to s3 data formats by @gloomweaver in https://github.com/spiceai/spiceai/pull/1414
• fix up dependencies now 0.5.0 disappeared by @Jeadie in https://github.com/spiceai/spiceai/pull/1417
• Add NSQL to FlightRepl by @Jeadie in https://github.com/spiceai/spiceai/pull/1409
• Update Cargo.lock by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1418
• Enable spice.ai replication for runtime.metrics table by @ewgenius in https://github.com/spiceai/spiceai/pull/1408
• Restructure the runtime struct to make it easier to test by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1420
• Make it easier to construct an App programatically by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1421
• Add an integration test for federation by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1426
• wait 2 seconds for the status to turn ready in refreshing status test by @y-f-u in https://github.com/spiceai/spiceai/pull/1419
• Add functional tests for federation push-down by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1428
• Enable push-down federation by default by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1429
• Add guides and examples about error handling by @ewgenius in https://github.com/spiceai/spiceai/pull/1427
• Add LRU cache support for http-based queries by @sgrebnov in https://github.com/spiceai/spiceai/pull/1410
• Update README.md - Remove bigquery from tablet of connectors by @digadeesh in https://github.com/spiceai/spiceai/pull/1434
• Update acknowledgements by @github-actions in https://github.com/spiceai/spiceai/pull/1433
• CLI wording and logs change reflected on readme by @y-f-u in https://github.com/spiceai/spiceai/pull/1435
• Add databricks_use_ssl parameter by @Sevenannn in https://github.com/spiceai/spiceai/pull/1406
• Update helm version and use v0.13.0-alpha by @Jeadie in https://github.com/spiceai/spiceai/pull/1436
• Don't include feature 'llms/candles' by default by @Jeadie in https://github.com/spiceai/spiceai/pull/1437
• Correctly map NullBuilder for Null arrow types by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1438
• Propagate object store error by @gloomweaver in https://github.com/spiceai/spiceai/pull/1415

Full Changelog: https://github.com/spiceai/spiceai/compare/v0.12.2-alpha...v0.13.0-alpha

Resources​

• Getting started with Spice.ai
• Documentation

Community​

Spice.ai started with the vision to make AI easy for developers. We are building Spice.ai in the open and with the community. Reach out on Slack or by email to get involved.

• Twitter: @spice_ai
• Slack: spiceai.org/slack
• Telegram: Spice AI Discussion
• Reddit: https://www.reddit.com/r/spiceai
• Email: hey@spice.ai

Announcing the release of Spice.ai v0.6.1-alpha! 🌶

Building upon the Apache Arrow support in v0.6-alpha, Spice.ai now includes new Apache Arrow data processor and Apache Arrow Flight data connector components! Together, these create a high-performance bulk-data transport directly into the Spice.ai ML engine. Coupled with big data systems from the Apache Arrow ecosystem like Hive, Drill, Spark, Snowflake, and BigQuery, it's now easier than ever to combine big data with Spice.ai.

And we're also excited to announce the release of Spice.xyz! 🎉

Spice.xyz is data and AI infrastructure for web3. It’s web3 data made easy. Insanely fast and purpose designed for applications and ML.

Spice.xyz delivers data in Apache Arrow format, over high-performance Apache Arrow Flight APIs to your application, notebook, ML pipeline, and of course through these new data components, to the Spice.ai runtime.

Read the announcement post at blog.spice.ai.

New in this release​

• Adds Apache Arrow Data Processor
• Adds Apache Arrow Flight Data Connector

Now built with Go 1.18.

Dependency updates​

• Updates to React 18
• Updates to CRA 5
• Updates to Glide DataGrid 4
• Updates to SWR 1.2
• Updates to TypeScript 4.6

Resources​

• Getting started with Spice.ai
• Documentation
• Quickstarts and Samples

Community​

Spice.ai started with the vision to make AI easy for developers. We are building Spice.ai in the open and with the community. Reach out on Slack or by email to get involved. We will also be starting a community call series soon!

• Twitter: @spice_ai
• Slack: spiceai.org/slack
• Telegram: Spice AI Discussion
• Reddit: https://www.reddit.com/r/spiceai
• Email: hey@spice.ai