Category: Programming

Article about programming

Airflow using with HDFS and Apache Spark

Posted on by maximus

Using Apache Airflow with Apache Spark and HDFS typically involves setting up a workflow where Airflow orchestrates the execution of Spark jobs, and the data processed by Spark is stored or retrieved from HDFS. Here’s an overview of how to set up such a system:

Airflow setup

Airflow is used to schedule and orchestrate the workflow. You’ll need to have Apache Airflow installed and configured with the necessary components for your Spark jobs.

Spark setup

Spark needs to be set up in a cluster environment (e.g., standalone mode, YARN, or Kubernetes). The Spark jobs will typically read and write data to/from HDFS.

HDFS setup

HDFS (Hadoop Distributed File System) serves as the distributed storage system where data is stored and retrieved by Spark jobs.

Airflow and Spark Integration

We need a Scala-based Spark application that will read/write data to/from HDFS or perform any transformation required. We will then submit this application using Airflow’s SparkSubmitOperator.

Here’s an example of a simple Scala-based Spark application that reads data from HDFS, processes it, and saves it back to HDFS.

import org.apache.spark.sql.SparkSession

object SparkJob {
  def main(args: Array[String]): Unit = {
    // Initialize Spark session
    val spark = SparkSession.builder
      .appName("Scala Spark Job")
      .config("spark.hadoop.fs.defaultFS", "hdfs://namenode_host:8020")
      .getOrCreate()

    // Read data from HDFS
    val inputData = spark.read.text("hdfs://namenode_host:8020/input_data.txt")

    // Perform transformation (example: convert to uppercase)
    val transformedData = inputData.rdd.map(line => line.getString(0).toUpperCase())

    // Save transformed data back to HDFS
    transformedData.saveAsTextFile("hdfs://namenode_host:8020/output_data")
    
    spark.stop()
  }
}

To build the Scala application we use a build tool like SBT (Scala Build Tool) to package the Scala application as a JAR file.

name := "SparkJob"

version := "1.0"

scalaVersion := "2.12.10"

libraryDependencies += "org.apache.spark" %% "spark-core" % "3.1.2"
libraryDependencies += "org.apache.spark" %% "spark-sql" % "3.1.2"

Then build the JAR file with the following command:

sbt package

This will create a JAR file under the target/scala-2.12/ directory, e.g., target/scala-2.12/sparkjob_2.12-1.0.jar.

In next step we need to install Apache Airflow and configure the necessary connections (like spark_default connection) to submit the Spark job to our cluster. Once the Scala Spark job is ready and packaged, we can submit it to our Spark cluster using the SparkSubmitOperator in Airflow. Here is how you can do that:

from airflow import DAG
from airflow.providers.apache.spark.operators.spark_submit import SparkSubmitOperator
from datetime import datetime

default_args = {
    'owner': 'airflow',
    'start_date': datetime(2025, 6, 2),
}

dag = DAG(
    'scala_spark_hdfs_etl',
    default_args=default_args,
    schedule_interval=None,
)

spark_submit_task = SparkSubmitOperator(
    task_id='submit_spark_job',
    conn_id='spark_default',  # Airflow Spark connection ID
    application='/path/to/target/scala-2.12/sparkjob_2.12-1.0.jar',  # Path to the Scala JAR file
    name='scala-spark-job',
    conf={'spark.hadoop.fs.defaultFS': 'hdfs://namenode_host:8020'},
    dag=dag,
)

spark_submit_task

Explanation:

  • application: Path to the JAR file generated by your Scala application.
  • conn_id: Connection ID for Spark. You may need to configure this in Airflow (this typically points to your Spark cluster, either YARN or standalone).
  • conf: Spark configuration, such as the HDFS URI.
  • task_id: Unique identifier for the Airflow task.

Airflow Workflow

Now that the DAG is set up, Airflow will submit the Spark job (written in Scala) to your Spark cluster. The flow will execute the Spark job as part of the DAG.

  • Scheduling: You can set up a schedule for when to run the Scala-based Spark job.
  • Monitoring: Airflow will monitor the task and show logs, task success/failure, and retries in the UI.

Optional – Handle HDFS Files in Scala:

If you need to interact with HDFS directly within your Scala Spark job (for example, listing files in HDFS or checking for certain files), you can use the HDFS API in Spark. Here’s an example to list HDFS files in your job:

val hdfsPath = "hdfs://namenode_host:8020/input_data/"
val fs = org.apache.hadoop.fs.FileSystem.get(new java.net.URI(hdfsPath), spark.sparkContext.hadoopConfiguration)
val status = fs.listStatus(new org.apache.hadoop.fs.Path(hdfsPath))

status.foreach(file => println(file.getPath))

Summary:

  1. Scala Application: Write your Spark job in Scala and package it as a JAR.
  2. Airflow DAG: Use the SparkSubmitOperator to submit your Scala-based Spark job.
  3. HDFS: Use HDFS paths and configurations in your Spark job to read/write data.
  4. Scheduling and Monitoring: Airflow handles scheduling and monitoring the execution of your Spark jobs.

Apache Iceberg and time travel queries with Scala/Spark

Posted on by maximus

Iceberg provides time travel functionality, allowing you to query the state of a table at a previous point in time. You can query the table as it existed at a specific snapshot or as of a certain timestamp.

Example: querying an Iceberg table at a specific snapshot

Each time you make changes (like insert or update) to an Iceberg table, a snapshot is created. You can query data from a particular snapshot.

def timeTravelBySnapshot(spark: SparkSession, snapshotId: Long): Unit = {
  val timeTravelDF = spark.read
    .option("snapshot-id", snapshotId)
    .table("spark_catalog.default.iceberg_table")
  
  timeTravelDF.show()
  println(s"Data retrieved for snapshot: $snapshotId")
}

We can find the snapshot ID using:

val historyDF = spark.sql("SELECT * FROM spark_catalog.default.iceberg_table.history")
historyDF.show()

Once we have the snapshot ID, pass it to timeTravelBySnapshot() to query the data as of that snapshot.

Example: querying an Iceberg table as of a timestamp

def timeTravelByTimestamp(spark: SparkSession, timestamp: String): Unit = {
  val timeTravelDF = spark.read
    .option("as-of-timestamp", timestamp)
    .table("spark_catalog.default.iceberg_table")
  
  timeTravelDF.show()
  println(s"Data retrieved as of timestamp: $timestamp")
}
// Example timestamp format: "2024-10-16T15:00:00.000Z"
// To execute:
timeTravelByTimestamp(spark, "2024-10-16T15:00:00.000Z")

In this way we can to handle time travel queries using Apache Iceberg and Spark.

How to create neural network with Scala?

Posted on by maximus

Creating a neural network in Scala typically involves either building the network from scratch using linear algebra or leveraging libraries like Breeze or DeepLearning4j (which has Scala bindings). Here’s a basic outline of how you could create a neural network in Scala:

Dependencies

First, we’ll need some libraries for mathematical operations. Using sbt (Simple Build Tool) we can add this to our build.sbt file:

libraryDependencies ++= Seq(
  "org.scalanlp" %% "breeze" % "1.2",
  "org.deeplearning4j" % "deeplearning4j-core" % "1.0.0-beta7"
)

Creating a neural network from scratch

Here’s a very simplified feed-forward neural network (without any optimization or training) built from scratch using Breeze for matrix operations:

  • Input layer, weights, and biases initialization – we need a way to store the weights and biases between layers.
  • Activation functions –implement a simple activation function (e.g., sigmoid).
  • Feedforward – implement the feedforward process where the input data is passed through the network.
  • Backpropagation (if we need training).
import breeze.linalg._
import breeze.numerics._
// Activation function (sigmoid)
def sigmoid(x: DenseMatrix[Double]): DenseMatrix[Double] = {
  1.0 /:/ (exp(-x) + 1.0)
}
// Derivative of sigmoid
def sigmoidDerivative(x: DenseMatrix[Double]): DenseMatrix[Double] = {
  x :* (1.0 - x)
}
// Feedforward step
def feedForward(input: DenseMatrix[Double], weights: Seq[DenseMatrix[Double]], biases: Seq[DenseMatrix[Double]]): Seq[DenseMatrix[Double]] = {
  var activations = Seq(input)
  for (i <- weights.indices) {
    val z = (weights(i) * activations.last) + biases(i)
    val a = sigmoid(z)
    activations = activations :+ a
  }
  activations
}
// Backpropagation (simplified version for gradient calculation, not full)
def backpropagation(expected: DenseMatrix[Double], activations: Seq[DenseMatrix[Double]], weights: Seq[DenseMatrix[Double]]): (Seq[DenseMatrix[Double]], Seq[DenseMatrix[Double]]) = {
  val error = activations.last - expected
  // Backpropagation steps here (update weights and biases)
  (weights, activations)
}
// Example usage
val inputLayerSize = 2
val hiddenLayerSize = 3
val outputLayerSize = 1
// Initialize weights and biases randomly
val weights1 = DenseMatrix.rand[Double](hiddenLayerSize, inputLayerSize)
val weights2 = DenseMatrix.rand[Double](outputLayerSize, hiddenLayerSize)
val bias1 = DenseMatrix.rand[Double](hiddenLayerSize, 1)
val bias2 = DenseMatrix.rand[Double](outputLayerSize, 1)
// Input data
val X = DenseMatrix((0.0, 1.0), (1.0, 0.0)) // 2 input features
val Y = DenseMatrix(1.0, 0.0)               // Expected output
// Feedforward
val weights = Seq(weights1, weights2)
val biases = Seq(bias1, bias2)
val activations = feedForward(X, weights, biases)
// Print activations
activations.foreach(println)

The first step is importing the necessary libraries. Here, we use Breeze, a Scala library that provides powerful tools for numerical processing like linear algebra, which is essential for machine learning.

  • breeze.linalg._: This gives us tools for handling matrices and vectors (like DenseMatrix, which is similar to a 2D array or matrix),
  • breeze.numerics._: This provides basic mathematical functions like exp (exponentiation).

In this neural network, we use the sigmoid function as the activation function. The sigmoid squashes input values between 0 and 1, which is useful in controlling the output of each neuron in a neural network.

def sigmoid(x: DenseMatrix[Double]): DenseMatrix[Double] = {
  1.0 /:/ (exp(-x) + 1.0)
}

The sigmoid derivative is used for calculating the gradient during backpropagation. The derivative is needed to adjust weights and biases during training.

def sigmoidDerivative(x: DenseMatrix[Double]): DenseMatrix[Double] = {
  x :* (1.0 - x)
}

The feedforward process is where the input passes through the network layer by layer to compute the output. At each layer, we multiply the input by the weights, add the biases, and then apply the activation function (sigmoid in this case).

Steps in feedForward:

Input: Start with the input as the initial activation.

For Each Layer:

  • calculate the weighted sum (z = W * a + b), where W is the weight matrix, a is the activation from the previous layer, and b is the bias,
  • apply the sigmoid activation function to z to get the next activation,
  • add the new activation to the sequence of activations.

This is a placeholder for the backpropagation step. Backpropagation is a method used to calculate the gradient (partial derivatives) needed for adjusting the weights and biases during training, based on the difference between predicted and actual values.

Breakdown

Neural Network Structure:

  • Input layer size: 2 neurons (for 2 input features).
  • Hidden layer size: 3 neurons.
  • Output layer size: 1 neuron (since we are outputting 1 value).

Random Weights and Biases:

  • weights1: Matrix of random values for the first layer, with size [3, 2] (3 neurons, 2 inputs),
  • weights2: Matrix of random values for the second layer, with size [1, 3] (1 output, 3 neurons in the hidden layer),
  • bias1 and bias2: Random biases for the layers.

Input Data:

  • X represents 2 inputs: (0.0, 1.0) and (1.0, 0.0).
  • Y is the expected output: 1.0, 0.0.

Feedforward Execution:

  • feedForward(X, weights, biases) computes the activations of each layer.

Printing the Activations:

  • After passing through the network, the activations of each layer are printed.

Using libraries

If we use DeepLearning4j (DL4J), the code will be simpler since the library abstracts most of the math and logic. We can build a neural network like this:

import org.deeplearning4j.nn.conf._
import org.deeplearning4j.nn.conf.layers._
import org.deeplearning4j.nn.multilayer._
import org.nd4j.linalg.factory.Nd4j
import org.nd4j.linalg.dataset.api.iterator.DataSetIterator
import org.deeplearning4j.datasets.iterator.impl.IrisDataSetIterator
// Build the configuration
val conf: MultiLayerConfiguration = new NeuralNetConfiguration.Builder()
  .seed(123)
  .updater(new Nesterovs(0.1, 0.9))
  .list()
  .layer(0, new DenseLayer.Builder().nIn(4).nOut(3)
    .activation("relu")
    .build())
  .layer(1, new OutputLayer.Builder(LossFunctions.LossFunction.MCXENT)
    .activation("softmax")
    .nIn(3).nOut(3)
    .build())
  .build()
// Initialize the model
val model = new MultiLayerNetwork(conf)
model.init()
// Load dataset (Iris for simplicity)
val irisIter: DataSetIterator = new IrisDataSetIterator(150, 150)
// Train the model
for (i <- 0 until 1000) {
  model.fit(irisIter)
}
// Evaluate or test model

Summary

This code demonstrates how to build a simple neural network using the basic principles of feedforward and (partially) backpropagation. Here’s what happens:

  • The input data is passed through the network, layer by layer, with weights and biases applied.
  • The sigmoid activation function ensures the values are normalized between 0 and 1.
  • Although the backpropagation logic is not fully implemented, this structure would allow for training once it is in place.

This example is highly simplified, but it provides a foundation for building more complex networks with multiple hidden layers, different activation functions, and learning algorithms.

Apache Iceberg partitioning data with Scala/Spark

Posted on by maximus

Introduction

Partitioning helps improve query performance by reducing the amount of data scanned during read operations. Iceberg allows dynamic partitioning and supports various partition types such as identity, bucket, and truncate.

Example: partitioning data by a column (e.g., Age)

Let’s partition the table by a column like age (bucket partitioning).

def partitionTable(spark: SparkSession): Unit = {
  // Create a new partitioned table
  spark.sql(
    """
      |CREATE TABLE IF NOT EXISTS spark_catalog.default.iceberg_partitioned_table (
      | id BIGINT,
      | name STRING,
      | age INT
      |) USING iceberg
      | PARTITIONED BY (bucket(5, age))
      |""".stripMargin)
  println("Partitioned Iceberg table created!")
}
// To execute:
partitionTable(spark)

Here, we’ve partitioned the table using the age column into 5 buckets. This is useful for optimizing queries when filtering based on age.

Query example with partitioned table:

When you query partitioned data, Iceberg only scans the partitions relevant to the query:

def queryPartitionedTable(spark: SparkSession): Unit = {
  val partitionedDF = spark.sql("SELECT * FROM spark_catalog.default.iceberg_partitioned_table WHERE age > 25")
  partitionedDF.show()
}
// To execute:
queryPartitionedTable(spark)

Apache Hudi CDC (Change Data Capture) Use case with Scala/Spark

Posted on by maximus

Apache Hudi (Hadoop Upserts and Incrementals) integrates with Apache Spark, making it possible to perform Change Data Capture (CDC) using Spark and Scala. Hudi enables incremental data processing by tracking changes (inserts, updates, deletes) and efficiently managing these on top of distributed storage like HDFS or cloud stores (e.g., S3, GCS).

In many business scenarios, we need to capture changes (inserts, updates, and deletes) from a source system and apply them incrementally to your data lake. Hudi’s upsert and incremental queries are well-suited for this scenario.

Set up Apache Hudi and Spark dependencies:

libraryDependencies ++= Seq(
  "org.apache.hudi" %% "hudi-spark3-bundle" % "X.Y.Z",
  "org.apache.spark" %% "spark-core" % "3.3.0",
  "org.apache.spark" %% "spark-sql" % "3.3.0"
)

Replace X.Y.Z with the appropriate version of Hudi and ensure Spark is aligned with the same version.

Create Hudi Table in Spark:

To manage updates and inserts in a table (i.e., CDC), Hudi uses two main table types:

  • Copy on Write (COW): More write-intensive, but gives consistent reads.
  • Merge on Read (MOR): Faster writes but slightly complex reads due to incremental merges.

You can create a Hudi table using Spark DataFrame API. First, load your data into a DataFrame, and then apply Hudi configurations.

import org.apache.spark.sql.{SaveMode, SparkSession}
val spark = SparkSession.builder
  .appName("Hudi CDC Example")
  .config("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
  .getOrCreate()
// Example data in DataFrame (e.g., CDC data)
val df = spark.read.json("path_to_cdc_data")
// Hudi table write options
val hudiOptions = Map[String, String](
  "hoodie.table.name" -> "hudi_cdc_table",
  "hoodie.datasource.write.operation" -> "upsert",  // To handle CDC events
  "hoodie.datasource.write.recordkey.field" -> "primary_key_column",  // Primary key for CDC
  "hoodie.datasource.write.precombine.field" -> "timestamp_column",   // Helps resolve duplicate rows
  "hoodie.datasource.hive_sync.enable" -> "true",                     // Optional if syncing to Hive
  "hoodie.datasource.hive_sync.database" -> "hudi_db",
  "hoodie.datasource.hive_sync.table" -> "hudi_cdc_table",
  "hoodie.datasource.hive_sync.partition_fields" -> "partition_column",
  "hoodie.datasource.write.hive_style_partitioning" -> "true"
)
// Writing to Hudi table using DataFrame API
df.write.format("hudi")
  .options(hudiOptions)
  .mode(SaveMode.Append)  // Append or Overwrite as needed
  .save("path_to_hudi_table")

Read Incremental Data

After loading data incrementally using Hudi, you can read the changes (new inserts, updates, and deletes) since the last commit.

// Reading Hudi data incrementally from the latest commit
val incrementalOptions = Map(
  "hoodie.datasource.query.type" -> "incremental",
  "hoodie.datasource.read.begin.instanttime" -> "20230101000000" // Specify the commit timestamp to start from
)
val incrementalDF = spark.read.format("hudi")
  .options(incrementalOptions)
  .load("path_to_hudi_table")
incrementalDF.show(false)

Handle Updates and Deletes in CDC

Apache Hudi allows you to manage updates and deletes through the upsert operation. When upserting data, Hudi automatically checks for the primary key and handles whether it should insert, update, or delete the record.

Handling Deletes

For deletions, the key field should be provided, and the operation should be set to delete in Hudi write options.

val deleteOptions = Map[String, String](
  "hoodie.table.name" -> "hudi_cdc_table",
  "hoodie.datasource.write.operation" -> "delete",  // For deletes
  "hoodie.datasource.write.recordkey.field" -> "primary_key_column"
)
val deleteDF = // Your dataframe containing the primary keys to delete
deleteDF.write.format("hudi")
  .options(deleteOptions)
  .mode(SaveMode.Append)
  .save("path_to_hudi_table")

Key Configurations

  • hoodie.datasource.write.operation: Choose upsert, insert, or delete depending on your CDC operation,
  • hoodie.datasource.write.recordkey.field: Set this to the primary key column, which is used to identify unique rows,
  • hoodie.datasource.write.precombine.field: Used for deduplication in the case of multiple updates to the same record,
  • hoodie.datasource.query.type: Used when reading, with options like snapshot (full read) or incremental (read only changed records).

Advantages of Using Apache Hudi for CDC

  1. Efficient Storage: Hudi manages the data lifecycle by compacting files and removing unnecessary duplicates.
  2. Streaming Use Case: With Hudi, you can efficiently consume CDC streams (e.g., from Kafka) and apply updates.
  3. Incremental Processing: Hudi allows for reading incremental changes, which improves processing efficiency when only new or changed data needs to be processed.

Example with Spark Structured Streaming (CDC Streaming)

If you’re processing CDC events from a stream (like Kafka), you can combine Spark Structured Streaming with Hudi as follows:

val streamDF = spark
  .readStream
  .format("kafka")
  .option("kafka.bootstrap.servers", "localhost:9092")
  .option("subscribe", "cdc-topic")
  .load()
// Write stream to Hudi table
streamDF.writeStream
  .format("hudi")
  .options(hudiOptions)
  .outputMode("append")
  .start("path_to_hudi_table")

This allows for continuous ingestion of changes and real-time updates into your Hudi dataset.

Summary

By combining Apache Hudi with Spark in Scala, you can efficiently implement a CDC system that supports upserts, deletes, and real-time data ingestion with minimal overhead. Hudi simplifies the management of large-scale distributed datasets by offering incremental data processing and optimized storage.

The Circuit breaker pattern in Java

Posted on by maximus

The Circuit Breaker pattern is used to prevent an application from performing an operation that is likely to fail. In microservices architecture, it’s commonly used to handle fault tolerance when calling external services that may fail or take a long time to respond.

Imagine a scenario where a microservice (Service A) calls another service (Service B). If Service B is down or slow to respond, without a circuit breaker, Service A will continue making requests, which may lead to cascading failures or resource exhaustion. The Circuit Breaker pattern “opens” and stops calling the failing service for a period to allow it to recover, then “closes” after the service is healthy again.

Key states of Circuit Breaker:

  1. Closed: The circuit breaker allows requests to pass through as normal.
  2. Open: The circuit breaker prevents requests from being sent, returning a fallback response immediately.
  3. Half-Open: The circuit breaker allows a limited number of requests to check if the service has recovered.

Implementation

There are different ways to implement Circuit Breaker pattern. In this case we’ll use Resilience4j for implementing the Circuit Breaker in Spring Boot. Add the following dependency to your pom.xml:

<dependency>
    <groupId>io.github.resilience4j</groupId>
    <artifactId>resilience4j-spring-boot2</artifactId>
    <version>1.7.0</version>
</dependency>

Here’s an example of how we might use a RestTemplate to call another service:

import org.springframework.stereotype.Service;
import org.springframework.web.client.RestTemplate;
import org.springframework.beans.factory.annotation.Autowired;
@Service
public class ExternalService {
    @Autowired
    private RestTemplate restTemplate;
    public String callExternalService() {
        String url = "http://localhost:8081/external-api";
        return restTemplate.getForObject(url, String.class);
    }
}

This method will call an external API, but what happens if this API is down or slow? This is where the Circuit Breaker comes in.

To apply the Circuit Breaker, annotate the method that makes the external call using the @CircuitBreaker annotation.

import io.github.resilience4j.circuitbreaker.annotation.CircuitBreaker;
import org.springframework.stereotype.Service;
@Service
public class ExternalService {
    @Autowired
    private RestTemplate restTemplate;
    private static final String EXTERNAL_SERVICE = "externalService";
    @CircuitBreaker(name = EXTERNAL_SERVICE, fallbackMethod = "fallbackResponse")
    public String callExternalService() {
        String url = "http://localhost:8081/external-api";
        return restTemplate.getForObject(url, String.class);
    }
    public String fallbackResponse(Throwable t) {
        return "External Service is currently unavailable. Please try again later.";
    }
}

@CircuitBreaker(name = "externalService"): This specifies that Resilience4j will manage the circuit breaker with the name externalService.

fallbackMethod = "fallbackResponse": If the circuit breaker is in the OPEN state, or if the call to the external service fails, this fallback method will be executed.

Now we can configure the circuit breaker properties in our application.yml file.

resilience4j.circuitbreaker:
  instances:
    externalService:
      slidingWindowSize: 5
      failureRateThreshold: 50
      waitDurationInOpenState: 10000
      permittedNumberOfCallsInHalfOpenState: 3
  • slidingWindowSize: The number of calls to evaluate for the failure rate,
  • failureRateThreshold: The percentage of failures that trip the circuit breaker. In this case, if 50% of calls fail, the circuit opens,
  • waitDurationInOpenState: The amount of time (in milliseconds) the circuit breaker stays open before transitioning to half-open state,
  • permittedNumberOfCallsInHalfOpenState: The number of calls to allow in the half-open state to check if the external service has recovered.

How it works

  1. Initially, the circuit breaker is in the Closed state. All requests pass through to the external service.
  2. If the failure rate (e.g., timeouts or errors) exceeds the failureRateThreshold, the circuit breaker transitions to the Open state. In this state, all requests are blocked, and the fallback method is executed.
  3. After the waitDurationInOpenState, the circuit breaker transitions to the Half-Open state, allowing a limited number of requests to check if the service has recovered.
  4. If the service responds successfully during the Half-Open state, the circuit breaker transitions back to Closed. Otherwise, it reopens.

Summary

The Circuit Breaker pattern prevents continuous calls to a failing service, providing a graceful degradation (using the fallback method) while the service recovers. Using Resilience4j with Spring Boot simplifies the implementation, allowing you to control fault tolerance with minimal effort.

Java 17 LTS through the eyes of Scala developer

Posted on by maximus

Introduction

Java and Scala are both modern, powerful languages that offer a range of features for developers, and while they differ in their paradigms (Java being primarily object-oriented and Scala being a hybrid of object-oriented and functional programming), they do share some similarities, especially as both languages have evolved. As an experienced Java developer (from Java 7 to the latest) and also experienced Scala developer I will try to compare the two languages.

1. Immutability

Immutability in Java refers to the design of objects whose state cannot be modified after they are created. Once an immutable object is instantiated, its fields cannot be changed, which provides several advantages, including thread safety, simplicity, and easier debugging. The most well-known immutable class in Java is String, but developers can create their own immutable classes.

Java 17

public record Person(String name, int age) {}

Records automatically generate:

  • constructor,
  • getters,
  • equals(),
  • hashCode(),
  • toString().

However, records provide shallow immutability.

record Team(List<String> members) {}

The reference cannot change, but the list can.

Scala 2.13

case class Person(name: String, age: Int)

Case classes are immutable by convention.

val p = Person("John", 30)

Fields are val unless explicitly declared var.

Scala 3

Same concept, with additional compiler improvements.

2. Sealed Types

Sealed types restrict who may extend a hierarchy.

Java 17

sealed interface Shape
    permits Circle, Rectangle {}

Useful for exhaustive pattern matching.

Scala 2.13

sealed trait Shape
case class Circle(r: Double) extends Shape
case class Rectangle(w: Double,h: Double) extends Shape

All subclasses must be in the same source file.

Scala 3

Exactly the same idea.

Scala 3 enums automatically create sealed hierarchies.

3. Pattern Matching

Pattern matching allows branching based on structure, not just type.

Java 17

if (obj instanceof String s) {
    System.out.println(s.length());
}

Switch pattern matching was still a preview feature.

Scala 2.13

shape match {
    case Circle(r) => ...
    case Rectangle(w,h) => ...
}

Can match:

  • values
  • types
  • tuples
  • lists
  • nested objects
  • regex
  • custom extractors

Scala’s pattern matching is far more expressive.

4. Type Inference

Type inference lets the compiler determine types automatically.

Java

var list = List.of(1,2,3);

Only local variables.

Scala

val list = List(1,2,3)

Inference works for

  • methods
  • lambdas
  • generic types
  • pattern matching
  • collections

Scala 3 further improves inference.

5. Lambdas

Lambdas represent anonymous functions.

Java

list.stream()
    .map(x -> x * 2)

Requires functional interfaces.

Scala

list.map(_ * 2)

Functions are true objects.

6. Collections

Collections are central to functional programming.

Java

Uses

  • List
  • Set
  • Map
  • Stream

Streams are separate from collections.

Scala

Everything supports

map
flatMap
filter
collect
groupBy
partition
fold
reduce
scan

without converting to streams.

7. Functional Programming

Functional programming emphasizes pure functions, immutable data, and function composition.

Java supports FP, but Scala is designed around it.

Scala libraries:

  • Cats
  • Cats Effect
  • ZIO
  • FS2

make purely functional programming practical.

8. Optional vs Option

Both represent optional values.

Java

Optional<String>

Designed mainly for return types.

Scala

Option[String]

Behaves like a collection.

option.map(...)
option.flatMap(...)
option.filter(...)

9. Null Safety

Null references are a common source of runtime errors.

Java

String s = null;

Perfectly legal.

Scala 2

Prefer

Option[String]

instead of null.

Scala 3

With Explicit Nulls

String

and

String | Null

are different types.

Much safer.

10. Exceptions

Exception handling reports errors during execution.

Java

  • checked exceptions
  • unchecked exceptions

Compiler forces checked exceptions.

Scala

Only unchecked exceptions.

Functional libraries usually avoid exceptions entirely by using:

  • Try
  • Either
  • IO
  • ZIO

11. Algebraic Data Types (ADT)

ADTs model complex domains by combining product types (AND) and sum types (OR).

Java

sealed interface Expr {}

record Num(int value) implements Expr {}
record Add(Expr l, Expr r) implements Expr {}

Scala

sealed trait Expr

case class Num(v:Int) extends Expr
case class Add(l:Expr,r:Expr) extends Expr

Scala 3

enum Expr:
  case Num(v:Int)
  case Add(l:Expr,r:Expr)

Scala offers much more concise ADT support.

12. Higher-Order Functions

Higher-order functions either accept functions as parameters or return functions.

Java:

list.forEach(System.out::println);

Scala:

list.foreach(println)

Higher-order functions are fundamental in Scala.

13. Implicits

Implicits allow values or conversions to be supplied automatically by the compiler.

Scala 2:

implicit val ec = ExecutionContext.global

Scala 3 replaces this with clearer syntax:

given ExecutionContext = ExecutionContext.global

def run(using ExecutionContext)

Java has no comparable language feature.

14. Extension Methods

Extension methods add behavior to existing types without inheritance.

Java uses utility classes:

Collections.sort(list);

Scala 2

implicit class RichString(s: String)

Scala 3

extension (s: String)
  def hello = ...

This is now a first-class language feature.

15. Value Classes

Value classes wrap existing types without (in many cases) introducing runtime allocation.

Scala 2:

class UserId(val value: Long) extends AnyVal

Scala 3 favors opaque types:

opaque type UserId = Long

These provide type safety with effectively zero runtime overhead.

Java has no direct equivalent in Java 17 (Project Valhalla aims to introduce value objects in a future release).

16. Type System

A language’s type system determines how expressive and safe programs can be.

Java offers:

  • generics
  • wildcards
  • bounded types
  • sealed types

Scala additionally supports:

  • higher-kinded types
  • variance annotations
  • path-dependent types
  • abstract type members
  • type classes
  • dependent function types (Scala 3)
  • union types (Scala 3)
  • intersection types (Scala 3)
  • match types (Scala 3)
  • opaque types (Scala 3)

Scala’s type system is among the most expressive of mainstream programming languages.

17. Compilation Speed

Compilation speed affects developer productivity, especially in large codebases.

  • Java 17: Fastest compilation due to a simpler type system and mature incremental compilation.
  • Scala 2.13: Noticeably slower because the compiler performs advanced type inference and implicit resolution.
  • Scala 3: Improves compiler architecture and is often faster than Scala 2.13, though still generally slower than Java for comparable projects.

18. Learning Curve

The learning curve reflects how quickly developers become productive.

  • Java 17: Gentle learning curve. A developer can become productive quickly thanks to a relatively small language surface and consistent syntax.
  • Scala 2.13: Steep learning curve. Besides object-oriented programming, developers must understand functional programming concepts, advanced collections, implicits, and a sophisticated type system.
  • Scala 3: Still challenging, but easier than Scala 2.13. The redesigned syntax, given/using mechanism, extension methods, enums, and improved error messages reduce much of the accidental complexity while preserving Scala’s expressive power.

Overall comparison

  • Choose Java 17 if you value simplicity, fast compilation, broad industry adoption, and ease of maintenance.
  • Choose Scala 2.13 if you work on existing Scala ecosystems (e.g., Spark, Akka Classic, Play Framework) and need maximum compatibility.
  • Choose Scala 3 if you want the full expressive power of Scala with a cleaner language design, stronger type safety, modern features such as enums, opaque types, union types, and native extension methods, and are starting a new Scala project.