Show HN: Denormalized – Embeddable Stream Processing in Rust and DataFusion https://ift.tt/LTNqPMF

Show HN: Denormalized – Embeddable Stream Processing in Rust and DataFusion tl;dr we built an embeddable stream processing engine in Rust using apache DataFusion, check us out at https://ift.tt/Y2X6IO7 Hey HN, We’d like to showcase a very early version of our embeddable stream processing engine called Denormalized. The rise of DuckDB has abundantly made it clear that even for many workloads of Terabyte scale, a single node system outshines the distributed query engines of previous generation such as Spark, Snowflake etc in terms of both performance and cost. Now a lot of workloads DuckDB is used for were normally considered to be “big data” in the previous generation, but no more. In the context of streaming especially, this problem is more acute. A streaming system is designed to incrementally process large amounts of data over a period of time. Even on the upper end of scale, productionized use-cases of stream processing are rarely performing compute on more than tens of gigabytes of data at a given time. Even so, the standard stream processing solutions such as Flink involve spinning up a distributed JVM cluster to even compute against the simplest of event streams. To that end, we’re building Denormalized designed to be embeddable in your applications and scale up to hundreds of thousands of events per second with a Flink-like dataflow API. While we currently only support Rust, we have plans for Python and Typescript bindings soon. We’re built atop DataFusion and the Arrow ecosystems and currently support streaming joins as well as windowed aggregations on Kafka topics. Please check out out repo at: https://ift.tt/Y2X6IO7 We’d love to hear your feedback. https://ift.tt/Y2X6IO7 August 16, 2024 at 12:16AM

Komentar

Posting Komentar

Postingan populer dari blog ini

Show HN: Interactive exercises for GNU grep, sed and awk https://ift.tt/OxeFwah

Show HN: Interactive exercises for GNU grep, sed and awk Hello! For the past few months, I've been using a Python framework called Textual to create TUI apps for interactive exercises. Released the app for GNU awk earlier today, so thought I'd create a post here. If you already know how to manage Python packages, you can use the following command to get all the three apps: pip install grepexercises sedexercises awkexercises `pipx` should also work, but I haven't tested it. The GitHub repo has the source code as well as more detailed installation instructions. You can use alternative CLI tools to solve these exercises as well. For example, Perl instead of GNU awk or ripgrep instead of GNU grep and so on. Hope you find these TUI apps useful. I'd highly appreciate your feedback. Happy learning :) https://ift.tt/Hp9kFEi August 17, 2023 at 05:13PM
Baca selengkapnya

Show HN: Create demos & guides just with a simple prompt https://ift.tt/HfWo3mz

Show HN: Create demos & guides just with a simple prompt Teams often create demos & guides on a daily basis. But creating them is time consuming and needs a lot of effort. From content to the branding, customizations, and many more - they are not something you would want to do on a daily basis. Enter Fable. Purpose built for teams in marketing, sales & customer success. Fable is apt for engaging visitors, closing deals faster, and onboarding your users quickly. https://ift.tt/IVCbkiG October 17, 2024 at 08:32PM
Baca selengkapnya