Apache Iceberg partitioning data with Scala/Spark

Introduction

Partitioning helps improve query performance by reducing the amount of data scanned during read operations. Iceberg allows dynamic partitioning and supports various partition types such as identity, bucket, and truncate.

Example: partitioning data by a column (e.g., Age)

Let’s partition the table by a column like age (bucket partitioning).

def partitionTable(spark: SparkSession): Unit = {
  // Create a new partitioned table
  spark.sql(
    """
      |CREATE TABLE IF NOT EXISTS spark_catalog.default.iceberg_partitioned_table (
      | id BIGINT,
      | name STRING,
      | age INT
      |) USING iceberg
      | PARTITIONED BY (bucket(5, age))
      |""".stripMargin)
  println("Partitioned Iceberg table created!")
}
// To execute:
partitionTable(spark)

Here, we’ve partitioned the table using the age column into 5 buckets. This is useful for optimizing queries when filtering based on age.

Query example with partitioned table:

When you query partitioned data, Iceberg only scans the partitions relevant to the query:

def queryPartitionedTable(spark: SparkSession): Unit = {
  val partitionedDF = spark.sql("SELECT * FROM spark_catalog.default.iceberg_partitioned_table WHERE age > 25")
  partitionedDF.show()
}
// To execute:
queryPartitionedTable(spark)