Spring Cloud Data Flow

2. Spring Cloud Data Flow Overview

Spring Cloud Data Flow (SCDF) is a microservices-based orchestration service for data integration and real-time data processing. It allows you to:

• Deploy streaming and batch data pipelines.
• Compose microservices (sources → processors → sinks).
• Monitor and manage applications at runtime.

Key concepts:

• Source: Produces data (e.g., HTTP, Kafka, File).
• Processor: Transforms data (e.g., filter, enrich, aggregate).
• Sink: Consumes data (e.g., log, database, Kafka).

SCDF uses Spring Boot apps under the hood.

3. Example: Streaming Pipeline with SCDF

Suppose we want to stream messages from HTTP → transform → log.

Step 1: Start SCDF

• You can run SCDF locally using Docker:

docker run -p 9393:9393 --name scdf -d springcloud/spring-cloud-dataflow-server

docker run -p 9393:9393 --name scdf -d springcloud/spring-cloud-dataflow-server

• Access the dashboard: http://localhost:9393/dashboard

Step 2: Create Microservices

Spring provides prebuilt apps:

• HTTP Source: receives HTTP POST requests.
• Transform Processor: modifies messages.
• Log Sink: logs messages to the console.

We can define the stream via SCDF shell or dashboard:

dataflow:>stream create --name mystream --definition "http | transform --expression=payload.toUpperCase() | log" --deploy

dataflow:>stream create --name mystream --definition "http | transform --expression=payload.toUpperCase() | log" --deploy

• http: accepts HTTP messages.
• transform: converts payload to uppercase.
• log: prints message to console.

Send data:

curl -X POST -d "hello world" http://localhost:8080

curl -X POST -d "hello world" http://localhost:8080

Output in log sink:

HELLO WORLD

HELLO WORLD

Step 3: Optional: Custom Spring Boot Apps

You can also create custom Spring Boot apps for SCDF.

Processor Example (UppercaseProcessor.java)

package com.example.demo;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.cloud.function.adapter.aws.SpringBootRequestHandler;
import java.util.function.Function;

@SpringBootApplication
public class UppercaseProcessorApplication {
    public static void main(String[] args) {
        SpringApplication.run(UppercaseProcessorApplication.class, args);
    }

    public Function<String, String> uppercase() {
        return value -> value.toUpperCase();
    }
}

package com.example.demo;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.cloud.function.adapter.aws.SpringBootRequestHandler;
import java.util.function.Function;

@SpringBootApplication
public class UppercaseProcessorApplication {
    public static void main(String[] args) {
        SpringApplication.run(UppercaseProcessorApplication.class, args);
    }

    public Function<String, String> uppercase() {
        return value -> value.toUpperCase();
    }
}

Then you register this app in SCDF and use it in a stream.

Summary