#distributed #query #data #processing #sql

bin datafusion

DataFusion is a SQL parser, planner, and execution framework for Rust with support for CSV and Apache Parquet file formats

31 releases

0.3.2 Aug 8, 2018
0.3.1 Jul 3, 2018
0.3.0-alpha.0 Jun 29, 2018
0.2.2 Mar 26, 2018

#39 in Database interfaces

Download history 19/week @ 2018-05-16 67/week @ 2018-05-23 137/week @ 2018-05-30 8/week @ 2018-06-06 38/week @ 2018-06-13 35/week @ 2018-06-20 345/week @ 2018-06-27 125/week @ 2018-07-04 40/week @ 2018-07-11 103/week @ 2018-07-18 155/week @ 2018-07-25 126/week @ 2018-08-01 121/week @ 2018-08-08

439 downloads per month

DataFusion: SQL Query Execution in Rust

License Version Build Status Coverage Status Gitter chat

DataFusion is a SQL parser, planner, and query execution library for Rust. A DataFrame API is also provided.

The following features are currently supported:

  • SQL Parser, Planner and Optimizer
  • DataFrame API
  • Columnar processing using Apache Arrow
  • Support for local CSV and Apache Parquet files
  • Single-threaded execution of SQL queries, supporting:
    • Projection
    • Selection
    • Scalar Functions
    • Aggregates (Min, Max, Count)
    • Grouping
  • User-defined Scalar Functions (UDFs)

DataFusion can be used as a crate dependency in your project to add SQL support for custom data sources.

A Docker image is also available if you just want to run SQL queries against your CSV and Parquet files.

I have plans to make DataFusion a fully distributed compute platform with features similar to Apache Spark, but I need help from contributors to get there.

Project Home Page

The project home page is now at https://datafusion.rs and contains the roadmap as well as documentation for using this crate. I am using GitHub issues to track development tasks and feedback.

Prerequisites

  • Rust nightly (required by parquet-rs crate)

Building DataFusion

See BUILDING.md.

Gitter

There is a Gitter channel where you can ask questions about the project or make feature suggestions too.

Contributing

Contributors are welcome! Please see CONTRIBUTING.md for details.

Apache-2.0 license

Dependencies

Reverse deps