Receiving a stream of events is the easy part. Point an Event Hub at your devices or your application, and telemetry pours in — thousands of messages a second if you like, neat and continuous. The trouble starts the moment someone wants to use it. Because a firehose of raw events is not something an analyst can query, a dashboard can sit on, or a business can make a decision from. It's a river, and what people actually need is a reservoir — something at rest, shaped, and queryable. The gap between "the data is streaming in" and "someone can answer a question with it" is where a startling number of real-time projects quietly stall, and having built a few smart-building platforms on exactly this problem, I want to lay out the pattern that bridges it.
Why raw streams resist analysis
An event stream and an analytical query want opposite things. A stream is unbounded, ordered by arrival, and optimised for movement — get each message from here to there, fast. An analytical query wants data at rest, organised for retrieval — give me all of yesterday's readings for this sensor, aggregated by hour. You can't simply run a warehouse query against a moving stream; the shapes fight each other. So the job is to land the stream into a form the warehouse world understands, without losing the timeliness that made it worth streaming in the first place. That balance — fresh and queryable — is the whole design problem.
The pattern: ingest, process, land, serve
The architecture that reliably works has four stages, and the discipline is keeping them cleanly separated:
- Ingest with something built to absorb the firehose. An Event Hub (or IoT Hub for device telemetry) is the front door — it takes the flood, buffers it, and hands it on without dropping messages when things spike. Don't try to process at the door; just catch everything reliably.
- Process in flight with a stream processor. Azure Stream Analytics sits on the hub and does the in-motion work: filtering noise, aggregating into time windows, computing running values, joining reference data. This is where a river of raw pings becomes "average occupancy per floor per five minutes" — the shaping that makes the data mean something.
- Land it in two places, on purpose. Here's the move people miss: send the processed output to both a hot path and a cold path. The hot path — a live dashboard, an alerting sink — gets near-instant data for the "what's happening right now" questions. The cold path lands the same data into the lake or warehouse, where it accumulates into the history that "how did this trend over the last month" questions need. Same stream, two destinations, two very different jobs.
- Serve from the landed data. Analysts and dashboards query the reservoir — the warehouse tables or lake files — not the live stream. They get the tidy, at-rest, queryable shape they need, kept fresh by the pipeline feeding it continuously.
The details that decide whether it works
- Window your aggregations deliberately. Streaming aggregation happens over time windows — tumbling, hopping, sliding — and choosing the right one is choosing what your metric means. A tumbling five-minute window and a sliding one answer subtly different questions; pick on purpose, not by default.
- Plan for late and out-of-order events. In the real world, especially with devices, messages arrive late, out of sequence, or duplicated. A robust pipeline has an explicit policy for lateness — how long to wait, what to do with stragglers — rather than pretending the stream is perfectly ordered. It isn't, and the pretence is where bad numbers come from.
- Don't over-refine the cold path. It's tempting to pre-aggregate everything before landing it in the lake, but the historical store is more useful with a fairly granular record you can re-aggregate later as questions change. Refine for the hot path; keep the cold path flexible.
- Watch the cost of always-on. A streaming pipeline runs continuously by nature, so its cost is continuous too. Size the stream processor to the real throughput and resist provisioning for a peak that never comes.
A concrete example, to make it real
Take a smart building. Occupancy sensors on every floor emit a reading every few seconds — thousands of tiny messages an hour, individually meaningless. They land in an IoT Hub. A Stream Analytics job aggregates them into "average occupancy per floor per five-minute window," and forks the output: the hot path drives a live facilities dashboard showing which floors are busy right now, while the cold path lands the same five-minute aggregates into the lake, where months of them accumulate. The facilities manager watching the live view and the analyst asking "how has Tuesday occupancy trended since we changed the layout" are served by the same pipeline — but one drinks from the stream and the other from the reservoir. Neither could do the other's job, which is exactly why you build both paths.
The lesson under it
What makes streaming projects succeed is rarely the streaming technology itself — the tools for ingesting and processing events are mature and mostly do what they say. What makes them succeed is respecting the boundary between data in motion and data at rest, and building the deliberate bridge between them rather than hoping a dashboard can drink from the firehose directly. The organisations that get real value from real-time data are the ones that landed it somewhere queryable, kept it fresh, and gave their analysts a reservoir instead of a river. The stream was never the point. What you could ask of it, once it stopped moving, always was.