Iceberg, The Right Idea - The Wrong Spec - Part 2 of 2: The Spec
Let us finally look at what is so wrong with the Iceberg spec and why this simply isn't a serious attempt at solving the metadata problem of large Data Lakes.
Posts with tag: data-structures
Iceberg, The Right Idea - The Wrong Spec - Part 1 of 2: History
Iceberg: The great unifying vision finally allowing us to escape the vendor lock-in of our database engines. One table and metadata format to find them ... And in the darkness bind them!
I love the idea! But I loathe the spec.
In this post, I’ll explain why you should be deeply skeptical of Iceberg as it exists today. I’ll argue that its design flaws are so severe, we're watching a new HADOOP-style disaster unfold in real time.
My reaction is visceral, but not simple. It requires a historical lens, which is why we must split this into a historical and a current day part. (Also, splitting it gives LinkedIn followers time to bring out their pitchforks)
Why are Databases so Hard to Make? - High Speed DML
After a brief intermezzo about testing (read about my thoughts here: Testing is Hard and we often use the wrong Incentives) - it is time to continue our journey together to where we will explore databases and all the wonderful things they can do.
To fully enjoy this blog entry, it will be useful if you first read the previous posts:
- Why are Databases so hard to Make? Part 1 - CPU usage
- Why are Databases so Hard to Make? Part 2 – Logging to Disk
Let's get to it...