Apache Iceberg and Trino: Powering Data Lakehouse Architecture

Apache Iceberg is a cornerstone of any open data lakehouse, providing the transactional foundation upon which highly scalable and flexible analytics can flourish. Along with Trino, it can be used to build a robust, scalable, and high-performance data lakehouse.

- Shivam Maurya

Over the past ten years, the emergence of Big Data has transformed how organisations store and process their data. The performance and reliability of traditional data warehouses lacks flexibility and cost-effectiveness. At the same time, data lakes have scale and affordability but are challenged with governance, schema enforcement, and performance limitations.

Enter the data lakehouse -- a new data architecture that combines the scale-out store of data lakes with the transactional and governance features of data warehouses. By allowing SQL workloads natively on object storage with capabilities such as ACID transactions, schema evolution, and time-travel queries, the lakehouse offers a single platform for BI, data science, and real-time analytics.

What is a data lakehouse?

A data lakehouse integrates the best practices of data lakes and data warehouses, filling the gap between scalable, flexible storage and transactional, dependable analytics. It provides an integrated platform where organisations can oversee the entire lifecycle of the data — from ingestion and processing to analytics and machine learning.

Traditional data lakes are planned for raw, bulk storage but do not include the capabilities necessary for enterprise-level analytics, including schema enforcement, data versioning, and ACID guarantees. Data warehouses have these features but are expensive, inflexible, and usually associated with proprietary vendors.

The lakehouse resolves this trade-off by putting warehouse-like capability into open data structures (such as Parquet and ORC), while keeping the scalability and economics of object storage systems (such as S3, HDFS, or GCS).

Apache Iceberg: Modern table format for the lakehouse