E

ETL

High-throughput asynchronous ETL and stream processing pipeline in Python

This project/library contains common elements related to ETL processes...

Configuration and data workflows for an instance of Apache Airflow for the DDRplatform

A self-service data ingestion platform that lets teams connect to 30+ databases (PostgreSQL, ClickHouse, Sybase, Snowflake, and more), upload CSVs with a visual column mapper, and execute bulk INSERT/UPSERT operations with real-time execution analytics.

Built on a FastAPI backend with Google OAuth, encrypted credential storage, and a full audit trail — paired with a React/TypeScript frontend that includes an AI-powered query risk analyzer before execution.

Designed for data engineering workflows where traceability and security matter: every operation is hashed into an immutable audit chain, and all inputs are validated against SQL injection at the API layer.

Rustroid Sentinel is a high-performance backend system built to directly integrate with the NASA NeoWs API, continually extracting and analyzing data on potentially hazardous near-Earth objects (NEOs). Built entirely in Rust using the tokio ecosystem, the service delivers a lightning-fast, highly concurrent data pipeline designed for demanding operational environments.

Public portfolio demo of a CRM analytics pipeline using Bitrix-style exports, Python ETL, DuckDB, PostgreSQL, and machine learning for deal probability scoring.

Mapping-config driven data migration tool for CSV/JSON/XML/TSV with validation and reporting.

ETL pipeline project using Python, pandas and SQLite to import and structure CSV, Excel and HTML data files.

Automated Dataset Creation & Publishing Pipeline - Scrape, clean, transform, validate and publish datasets to HuggingFace Hub

https://ayinedjimi-consultants.fr

End-to-end AWS data lake pipeline for fleet telemetry data using S3, Spark, and Athena. Includes partitioned Parquet ETL, vehicle safety analytics, and SQL queries for overspeed and harsh braking detection.

This is a study project built using FastAPI to practice microservice architecture, data normalization techniques, and clean API design.

The service receives raw payloads from different simulated sources and transforms them into a standardized and validated structure.

It centralizes normalization logic and demonstrates how to build a scalable, maintainable, and test-friendly data processing layer.

Solución end-to-end para la migración y análisis de datos utilizando Python, FastAPI, Kafka y PostgreSQL. Implementa un pipeline de datos asíncrono y una API RESTful para analíticas, todo completamente containerizado con Docker Compose para un despliegue fácil y reproducible.

Unified project demonstrating both batch analytics and real-time streaming pipelines with Apache Spark:

Batch (PySpark/Jupyter): Processed S&P 500 stock data, applied transformations, and ran distributed computations.

Streaming (Spark + Kafka): Built a streaming pipeline to consume Kafka topics, process messages in real-time, and visualize outputs.

Deployed using Docker and Jupyter for reproducibility.

Analyzed decades of historical weather station data (1920–1940) using Hadoop MapReduce. Filtered operable stations, computed descriptive statistics (min, max, mean, median), and produced reports/graphs. Designed modular MRJobs to chain tasks together for scalable processing.

Advanced data synchronization framework.

Reporting for MIT Club of Northern California

В данном проекте находятся два задания, написанные на Python и реализующие выполнение цепочек задач (DAG) в среде Airflow

Simulation of a real hospital scenario with a ML model in production

Official weather data ETL for wind energy project evaluation in El Calafate, Argentina.

target-core is a Singer Target which intend to work with regular Singer Tap. The Goal is to use this package as a foundation to build other targets focusing on the core features, reducing the energy spent on maintaining the common parts.