sajjit.soti
← Back to projects

Case study — Personal SRE / security data project

Building a real-time SIEM pipeline: from “checking logs” to automated alerting

This project was built to get hands-on with Observability and Site Reliability Engineering, not just security tooling: a full event-driven pipeline that generates network logs, enriches them against a live threat feed in real time, and turns raw data into dashboards and alerting rules — not a one-off analysis script, but a system that watches itself.

Kafka

Real-time event streaming backbone

IPSum

Live threat-intel enrichment feed

Self-monitoring

“Pipeline down” health alert

Docker Compose

Fully reproducible, portable stack

01

The pipeline, end to end

A Python script generates thousands of simulated IDS-style network logs and streams them through Apache Kafka, which acts as the high-throughput message broker between producer and consumer. A second script picks up each log, checks the source IP against the live IPSum threat-intelligence feed, and pushes the enriched record into Elasticsearch. From there, Kibana is the command center — dashboards tracking total events and attack trends, with KQL queries to filter and investigate specific alerts. The whole stack runs on Docker, developed and tested from a Kali Linux environment.

02

From logs to alerts

Ingesting logs was the easy part — the useful part was configuring threshold rules in Kibana to detect “high severity spikes” as they happen, rather than requiring someone to go read through raw events after the fact. That’s the real shift this project was built around: moving from checking logs to automated alerting.

03

Building in resilience, not just detection

One of the more SRE-flavored additions was a “pipeline down” alert — the system watching its own health, not just the data flowing through it. In an SRE role, knowing the second a service fails, and seeing when it recovers, is the difference between a minor blip and an outage nobody notices until it’s too late.

04

Why Docker Compose

Managing Kafka, Elasticsearch, and Kibana as a multi-container architecture via Docker Compose — rather than installing each piece by hand — turned out to be one of the bigger practical lessons: a reproducible, portable environment matters as much as the pipeline logic itself, especially when the goal is a system someone else could actually stand back up.