Real-Time Weather Stream

I created an Event Hubs resource in an Azure resource group. Event Hubs is the ingestion layer, able to absorb high-throughput event streams in real time.

Events are synthetic, generated with the Event Hubs Data Explorer: JSON messages, each representing a single record as key-value pairs — like a row in a table.

2.2.png

A Databricks workspace is created in the same resource group, with a compute cluster to process the stream. The Azure Event Hubs connector library is installed on the cluster so that Spark Structured Streaming can read events efficiently.

2.1-2.png2.3.png

Medallion structure in Unity Catalog

In a Databricks notebook, I first create the catalog and the bronze, silver and gold schemas so that data is ingested, processed and refined in a structured way.

3.1.0.png

Connecting to Event Hubs

The connection is configured with the Event Hubs namespace, hub name and access key.

3.2-1.png

Reading the stream

Spark Structured Streaming then reads and displays the incoming events.

3.3.png3.4.png

Persisting to Delta

With readStream alone, data only lives in memory and disappears when the stream stops. Using writeStream, events are persisted to the Bronze layer in Delta format, so they remain available for processing and analysis.

4.1-1.png4.2-2.png

In Bronze, each event payload sits in a binary body column. To make it queryable, it has to be decoded and structured.

1. Binary to string

The body column is converted to a string with cast().

5.6.png5.7.png

2. Parsing the JSON

A schema is defined with PySpark StructType and StructField, then from_json() turns the string into structured columns.

5.2-1.png5.3.png

3. Writing the Silver table

writeStream saves the structured data to the Delta table streaming.silver.weather, processing it incrementally so new events keep flowing in without overwriting existing records.

5.4.png5.5.png

The Silver stream is aggregated over 5-minute windows to produce summarised metrics that are ready for reporting.

6.1-1.png

The aggregates are persisted to streaming.gold.weather_summary for efficient querying.

6.3-2.png

• →Event Hubs ingests real-time JSON events, streamed continuously into Databricks.
• →Bronze keeps the raw events in their original form.
• →Silver decodes binary records into structured JSON and extracts weather attributes such as temperature, humidity and wind speed.
• →Gold computes 5-minute averages for each metric, ready for visualisation.

Power BI connects through Databricks Partner Connect, which provides a connector bound to the cluster. Power BI queries the Gold layer directly, so reports stay up to date in near real time.

7.1-1.png

The report shows two views: the Silver layer with cleaned events, and the Gold layer with metrics aggregated over 5-minute intervals. Users filter by time and explore temperature by weather condition (Cloudy, Partly Cloudy, Sunny).

ffr.png