Kedify | Apache Kafka

Use events from Apache Kafka to trigger autoscaling with Kedify and KEDA

Apache Kafka is a distributed streaming platform that lets you publish, subscribe to, and process streams in real-time

It’s designed for high throughput and scalability, it is ideal for handling data feeds with high velocity and volume in fault-tolerant ways across distributed systems or applications.

Featured Use Cases

Scenario:

Organizations must process logs to ensure compliance with data protection regulations. These logs require specific handling or transformation, such as anonymization or encryption, at regular intervals.

Apache Kafka Usage:

Kafka collects and stores logs from various systems, aggregating them for compliance checks and transformations.

KEDA Usage:

KEDA triggers batch jobs to process sensitive log data, using Kafka lag to determine when to scale up processing capabilities. This ensures that logs are processed in compliance with legal time frames.

Get Started

                    apiVersion: keda.sh/v1alpha1
kind: ScaledJob
metadata:
  name: compliance-log-processing-job
  namespace: default
spec:
  jobTargetRef:
    template:
      spec:
        containers:
        - name: log-compliance-processor
          image: compliance-processor:latest
        restartPolicy: Never
  pollingInterval: 600  # Every 10 minutes
  successfulJobsHistoryLimit: 2
  failedJobsHistoryLimit: 3
  maxReplicaCount: 5
  triggers:
  - type: kafka
    metadata:
      bootstrapServers: kafka.svc:9092
      consumerGroup: compliance-log-group
      topic: compliance-log-topic
      lagThreshold: '50'

Scenario:

Analytics applications need to process data from IoT devices in real time to provide actionable insights for automated decision-making or alerting.

Apache Kafka Usage:

Kafka facilitates the high-throughput ingestion of streaming data from thousands of IoT devices, feeding into analytics applications.

KEDA Usage:

KEDA scales analytics applications dynamically, based on Kafka topic lag, ensuring minimal delay in data processing and timely delivery of insights.

Get Started

                    apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: streaming-data-analytics-scaledobject
  namespace: default
spec:
  scaleTargetRef:
    name: analytics-processing-deployment
  pollingInterval: 30
  minReplicaCount: 1
  maxReplicaCount: 20
  triggers:
  - type: kafka
    metadata:
      bootstrapServers: kafka.svc:9092
      consumerGroup: streaming-analytics-group
      topic: iot-data-topic
      lagThreshold: '10'
      offsetResetPolicy: latest

Scenario:

Applications in an event-driven architecture respond to events like user activities or system alerts, requiring timely processing to maintain system integrity and user experience.

Apache Kafka Usage:

Kafka acts as the backbone for event ingestion and distribution, ensuring that events are consistently published to the correct channels.

KEDA Usage:

KEDA helps maintain event processing speeds by scaling services based on the event queue length in Kafka topics, optimizing resource use and reaction times.

Get Started

                    apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: event-driven-architecture-scaledobject
  namespace: default
spec:
  scaleTargetRef:
    name: event-processing-deployment
  pollingInterval: 30
  minReplicaCount: 2
  maxReplicaCount: 20
  triggers:
  - type: kafka
    metadata:
      bootstrapServers: kafka.svc:9092
      consumerGroup: event-group
      topic: event-topic
      lagThreshold: '5'
      offsetResetPolicy: latest