Using Apache Iceberg for Developing Modern Data Tables

Apache Iceberg, a data management powerhouse, empowers businesses with its remarkable flexibility, seamless tool compatibility, and endorsement by the major tech players in the industry.

- Vinayak R. Adkoli

The cloud has made it possible to collect vast quantities of data and store it at a reasonable cost. It is also responsible today for the emergence of data lakes, data mesh and other modern data architectures. However, handling large amounts of data using a generic cloud offers its own challenges and limitations.

For instance, query engines often cannot react with typical blob storage systems smoothly. To address this issue, applying a table format to data becomes extremely useful.

Table format data was originally developed by Netflix and was open sourced as an Apache incubator project in 2018. It graduated from this in 2020 and became Apache Iceberg. Apache Iceberg can meet certain challenges faced by Apache Hive.

Process engines used by a company may change from time to time. For example, many business firms have moved from Hadoop to Spark or Trino. Iceberg provides greater flexibility and choice for processing engines.