K

kafka

Learning Spark in java/scala

Lab CDC event-driven : PostgreSQL → Debezium → Kafka → consumers Python (notifications, audit, risk scoring). Illustre WAL, replication slots, outbox pattern et eventual consistency sur un cas bancaire réel.

Projet de référence d'une architecture Lakehouse moderne appliquée à la détection de fraude bancaire.

Simule un environnement de production avec trois sources de données hétérogènes (fichiers CSV, base PostgreSQL, streaming Kafka/Redpanda) ingérées en continu vers un stockage objet S3-compatible (MinIO).

Stack technique :

Ingestion batch : Apache Spark (PySpark) + Delta Lake Ingestion streaming : Spark Structured Streaming + Redpanda (Kafka) Orchestration : Apache Airflow Transformation : dbt (DuckDB) Stockage : MinIO (S3), Delta Lake (Bronze/Silver), Parquet (Gold) Exploration : DuckDB / DBeaver

Architecture en médaillon (Medallion Architecture) :

Bronze : données brutes, sources séparées Silver : données nettoyées, déduplication inter-sources Gold : agrégats métier (fraude par heure)

L'ensemble de la stack tourne en local via Docker Compose.

Production-style real-time crypto streaming pipeline (Binance → Kafka → Polars Parquet → DuckDB → dbt) built locally with Docker and Python.

1000+ DevOps Bash Scripts - AWS, GCP, Kubernetes, Docker, CI/CD, APIs, SQL, PostgreSQL, MySQL, Hive, Impala, Kafka, Hadoop, Jenkins, GitHub, GitLab, BitBucket, Azure DevOps, TeamCity, Spotify, MP3, LDAP, Code/Build Linting, pkg mgmt for Linux, Mac, Python, Perl, Ruby, NodeJS, Golang, Advanced dotfiles: .bashrc, .vimrc, .gitconfig, .screenrc, tmux..

WebSocket proxy that allows clients to produce and consume messages from topics in Apache Kafka. Documentation can be found at https://kpmeen.gitlab.io/kafka-websocket-proxy/

Real-time stock market data lakehouse using Kafka, Bronze/Silver/Gold architecture, and Postgres feature serving.

Real-time connected vehicle telemetry pipeline using Kafka, MongoDB, FastAPI, and Grafana with anomaly detection and fleet monitoring dashboards.

-> Contenu -Un seul topic order-events sur lequel sont publiés trois types d’événements : OrderCreated, PaymentValidated, OrderShipped. -Producer (REST API) : publie les événements ; Consumer : écoute le topic avec trois consumer groups (groupIds distincts) et délègue au domaine via des handlers. -Architecture : hexagonale + template method ; contrats partagés dans le module kafka-contracts (OpenAPI). -> Stack: -Java 21, Spring Boot 3.2, Spring Kafka -Kafka (Zookeeper), Kafka UI -Maven multi-module : kafka-contracts, order-event-producer, order-event-consumer -Docker / Docker Compose pour l’infra et les apps -CI/CD GitLab : build Maven, tests unitaires, build & push des images sur Docker Hub, déploiement par SSH + docker compose sur le serveur. -> Dépôts d’images -Images poussées sur Docker Hub par la CI : kafka-kata-producer, kafka-kata-consumer.

Change data capture on Oracle databases + transfer of change events to Kafka

Streaming data pipeline for European Day-Ahead electricity market — Cats Effect 3 + fs2 + http4s + Kafka

Saga choreography pattern for managing distributed transactions across microservices using asynchronous events. Each service participates in the saga by emitting and responding to domain events, eliminating a central orchestrator. The project includes multiple cooperating services, event definitions, and examples of compensation flows to handle failures in event sequences.

Saga pattern with an orchestrator for managing long‑running distributed transactions. A central coordinator (orchestrator) directs microservices through a series of steps and compensation actions when necessary. It’s a learning lab designed to contrast with event‑driven choreography, show transaction boundaries, and explore rollback strategies in distributed systems.

Krun reads events from a Kafka topic and run a system command passing the read event to the stdin of the command. The command can be a bash script or anything else that can manage a stdin.

Sistema event-driven con Kafka que transforma documentos no estructurados en especificaciones de software completas. Extrae texto con OCR, procesa NER con transformers, clasifica oraciones y generar SRS en múltiples formatos.

E-commerce backend built with Java 17 + Spring Boot 3, Clean Architecture, Kafka, Elasticsearch, MySQL, JWT auth and observability (Filebeat + Elasticsearch).

Solución end-to-end para la migración y análisis de datos utilizando Python, FastAPI, Kafka y PostgreSQL. Implementa un pipeline de datos asíncrono y una API RESTful para analíticas, todo completamente containerizado con Docker Compose para un despliegue fácil y reproducible.

FastAPI that reads rss feeds from magazines, consolidates and broadcast them to pub/sub and clients.

A Spring Boot kata project for delivery modes and time slot booking (Drive, Delivery, Delivery Today, ASAP)