What I did

Big Data IoT Pipeline

Real-time IoT data processing with Kafka and PySpark

The pipeline reads IoT sensor data in real time. Kafka carries the sensor streams. PySpark processes the streams with custom user-defined functions (UDFs). The UDFs detect anomalies in the data. MongoDB stores the processed data. Matplotlib shows the results in charts.

PySparkKafkaMongoDBMatplotlib

View on GitHub →

Features

Kafka streams the raw sensor data.
PySpark processes the stream with custom user-defined functions.
The functions detect anomalies in temperature, humidity, and pressure.
Window functions compute rolling statistics.
MongoDB stores the processed data.
Matplotlib updates the charts when new data arrives.

How it works

Start the sensor simulator to publish data to Kafka.
Let PySpark read the raw stream.
Let the functions detect anomalies.
Let the pipeline write the data to MongoDB.
Watch the charts update in real time.

Tech stack

Streaming: Apache Kafka
Processing: PySpark, Spark Structured Streaming
Storage: MongoDB
Visualization: Matplotlib