Skip to content
The methodOpen data
Back to the collection

One SQL to Rule Them All

Edmon Begoli, Tyler Akidau, Fabian Hueske

Show all 6 authors

Julian Hyde, Kathryn Knight, Kenneth Knowles

2019 · arXiv

Databases
Read the paper
HN score
1
Mentions
4
Readers
4
Months
4
What’s in a score?

From the abstract

Real-time data analysis and management are increasingly critical for today`s businesses. SQL is the de facto lingua franca for these endeavors, yet support for robust streaming analysis and management with SQL remains limited. Many approaches restrict semantics to a reduced subset of features and/or require a suite of non-standard constructs. Additionally, use of event timestamps to provide native support for analyzing events according to when they actually occurred is not pervasive, and often comes with important limitations. We present a three-part proposal for integrating robust streaming into the SQL standard, namely: (1) time-varying relations as a foundation for classical tables as well as streaming data, (2) event time semantics, (3) a limited set of optional keyword extensions to control the materialization of time-varying query results. Motivated and illustrated using examples and lessons learned from implementations in Apache Calcite, Apache Flink, and Apache Beam, we show how with these minimal additions it is possible to utilize the complete suite of standard SQL semantics to perform robust stream processing.

Metadata: arXiv. Abstracts retain their original authors’ rights.

In the conversation

Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels

4 selected excerpts · 4 total mentions

cube2222
Recommendation · rule
Looking at the comments I see there are more and more such projects, that's great! Declarative interfaces rule! Anyways, I too felt a void to be filled with tools like these, so I (and a couple of other folks) developed OctoSQL[0], check it out if you like this. It lets you query json, csv, and various databases using SQL, and also let's you join between them. It differs from most of the other tools in that it also supports streaming data sources using temporal SQL extensions. (Inspired by the great paper, One SQL To Rule Them All[1]) [0]: https://github.com/cube2222/octosql [1]: https://arxiv.org/abs/1905.12133
Original comment Q – Run SQL Directly on CSV or TSV Files
BenoitP
Neutral mention · rule
> this would still leave the problem where each transactions causes two deletes and two inserts Indeed! You might be interested in the '6.5 Materialization Controls' section of this paper: https://arxiv.org/pdf/1905.12133.pdf They intend to add these extensions: SELECT ... EMIT STREAM AFTER DELAY INTERVAL '6' MINUTES ; -- to fix the output multiplication you mention SELECT ... EMIT STREAM AFTER WATERMARK ; -- this should fix the two deletes and two inserts I don't know if this is still in active development, though. There has been no news since the paper got published 2 years ago. > It doesn't seem like there is window you could set that would be guaranteed to connect…
Read the full comment
them? No there isn't. But I'm quite confident having a transaction-centric approach can be obtained in SQL. Your use case is perfect for illustrating the problem, still. I'd try something like: CREATE VIEW credits AS SELECT to_account AS account, sum(amount) AS credits, last_value(id) AS updating_tx FROM transactions GROUP BY to_account; And then try to join credits and debits together by updating_tx. No idea on how to check that accounts don't go negative, though.
Original comment Internal Consistency in Streaming Systems
james_woods
Neutral mention · rule
Where and how in dataflow is late data being handled? How can I configure in which ways refinements relate? These questions are the standard "What Where When How" I want to answer and put into code when dealing with streaming data. I was not able to find this in the documentation, but I only spent a few minutes scanning it. https://www.oreilly.com/radar/the-world-beyond-batch-streami... https://www.oreilly.com/radar/the-world-beyond-batch-streami... Also "Materialize" seems not to support needed features like tumbling windows (yet) when dealing with streaming data in SQL: https://arxiv.org/abs/1905.12133 Additionally "Materialize" states in their doc: State is all in…
Read the full comment
totally volatile memory; if materialized dies, so too does all of the data. - this is not true for example for Apache Flink which stores its state in systems like RocksDB. Having SideInputs or seeds is pretty neat, imagine you have two tables of several TiBs or larger. This is also something that "Materialize" currently lacks: Streaming sources must receive all of their data from the stream itself; there is no way to “seed” a streaming source with static data.
Original comment Why isn't differential dataflow more popular?
cangencer
Neutral mention · rule
It's not very different than normal SQL. Imagine that instead of a finite result set, you instead have a never ending result set. You can also roughly map operations like windowed aggregation into SQL with some additional syntax. This paper gives a pretty good overview, even though we don't fully agree with the model presented here: https://arxiv.org/abs/1905.12133
Original comment Show HN: Jet – in-memory, fault-tolerant, distributed stream processing

Keep following the thread.

More in Databases

More from the same topics.

2017arXiv

The Case for Learned Index Structures

Tim Kraska, Alex Beutel, Ed H. Chi +2

Indexes are models: a B-Tree-Index can be seen as a model to map a key to the position of a record within a sorted array, a Hash-Index as a model to map a key to a position of a record within an unsorted array, and a BitMap-Index as a model to indicate if a data record exists or not.

Databases
19 mentions14 months
5HN score