For the last stretch I've been building a data platform for smart buildings — turning a building full of sensors into something you can actually understand and act on — and it's been the most fun I've had in a while, partly because so little of it works the way the tutorials imply. So this is a field note on what it actually takes to build one of these from scratch: the shape of the architecture, why each piece is there, and the awkward realities the neat diagrams leave out.

Let me start with the shape, because once you see it, a lot of the individual decisions explain themselves.

The shape: two paths for the same data

The core insight that organises everything is that sensor data needs to go to two places at once, for two different reasons, and trying to serve both from one path is where designs go wrong.

  • The hot path is for now. A sensor says the temperature just spiked, or the room is suddenly occupied, and something may need to react to that within seconds. This path is about data in motion — processed as it arrives, feeding live dashboards and real-time alerts.
  • The cold path is for later. That same reading also needs to be stored, cheaply and durably, so that in three months you can ask "what's the occupancy pattern of this floor been all quarter?" This path is about data at rest — kept, accumulated, queried in bulk.

The same reading flows down both. The mistake I see — and nearly made — is trying to build one path that does both jobs, which produces something that's too slow to be real-time and too expensive to be an archive. Split them early.

The pieces, and why each one is there

With that shape in mind, here's roughly how the platform is assembled, and the why matters more than the names:

  1. The front door: IoT Hub. Every sensor connects here. IoT Hub is the managed entry point that handles thousands of devices talking at once, securely, without you building your own messaging infrastructure. It exists so that "a building's worth of devices connecting reliably at scale" is somebody else's solved problem, not yours.
  2. The splitter: stream processing. As data pours through the front door, a stream-processing layer reads it in motion — routing the urgent stuff down the hot path to react on, while everything also carries on down to storage. This is where "data in flight" becomes "decisions and archives."
  3. The hot path: live dashboards and alerts. Processed events feed something people watch and something that fires when a condition is met. This is the part that makes the building feel alive — that lets someone see, right now, what's happening across the floors.
  4. The cold path: cheap, durable storage. Every reading lands in low-cost storage designed to hold enormous volumes affordably, so history accumulates without the bill spiralling. Later, that's what you run the big analytical questions against.
  5. The reporting layer: Power BI. On top of the stored history sits the reporting people actually open — the trends, the comparisons, the "is this building performing better than that one" questions that only make sense over time.

None of these pieces is exotic on its own. The craft is in how they fit — and in the seams between them, which is exactly where the tutorials go quiet.

The parts the tutorials skip

Here's what nobody warns you about, learned the way you learn these things — by hitting them.

  • Sensors are liars, and you find out downstream. A tutorial's data is clean. A real building's sensors drop offline, report impossible values, get stuck repeating the last reading, and come back with a flood of backdated messages. Your platform's real job is less "move the data" and more "cope gracefully with data that's wrong," and that reality reshapes every layer. (I'll write more about handling lying sensors on its own, because it deserves it.)
  • Volume creeps, then leaps. One sensor is nothing. A building's worth, sampling every few seconds, every day, forever, is a genuinely large and relentless stream — it never pauses, never has a quiet night. Designs that are fine for a demo quietly buckle under the sheer continuousness of it, and you only find out weeks in.
  • The seams are where it breaks. Each individual service works. The failures live in the joins — the message that arrives late, the format that shifts, the reprocessing that double-counts. Most of my actual debugging time went not into any one component but into the handoffs between them.
Building the platform is assembling well-documented pieces, and that part is almost easy. The real work is everything that happens because the physical world feeding it is messy, relentless, and doesn't read the documentation.

The pieces the diagram doesn't show: identity and security

There's a whole layer the tidy architecture diagram omits, and it's the one that keeps me up at night more than throughput does: identity. A building full of sensors is a building full of things connecting to your cloud — and every one of them is a small door. Get device identity wrong and you've got either an unmanageable mess or a security hole with a physical footprint.

The lesson I learned quickly is that every device needs its own identity, not a shared key. It is so tempting, early on, to give every sensor the same credential, because provisioning is easier that way. It's also a design you deeply regret the first time you need to revoke one compromised device and discover you can't — because they all share a key, so revoking it locks out the entire building. Per-device identity means you provision more carefully up front and you can deal with a single bad actor without punishing everything else. IoT Hub supports this properly; use it, even though the shortcut beckons every single time.

Schema drift: the slow leak

One more from the school of hard knocks: the format of what your sensors send will change under you, and rarely with warning. A firmware update alters a field. A new batch of devices reports subtly differently. A vendor "improves" their payload. Each change is small; the cumulative effect, if you've assumed a fixed schema everywhere, is a platform that silently starts dropping or misreading data because reality drifted away from the shape your code expects.

The defence is to treat the incoming message format as something that will change rather than something fixed — validating at the boundary, and being loud when something arrives that doesn't fit, instead of silently swallowing it. A platform that shouts "I got a message I didn't understand" is mildly annoying. One that quietly accepts and mis-stores it is genuinely dangerous, because you find out months later that a whole floor's worth of data has been subtly wrong the entire time, and now every conclusion built on it is suspect.

A single reading's journey through the platform

Let me trace one temperature reading from a sensor on the second floor, all the way through, because following one datum end-to-end shows how the pieces actually interlock better than any diagram.

The sensor takes the reading and sends it to IoT Hub, authenticating with its own device identity — so before the data is even data, the platform knows which device this is and that it's allowed to speak. The reading lands in the hub as a message, one of thousands arriving that second, and the hub holds it, ordered, ready to be read.

From there the stream processor picks it up in flight. This is the fork: the reading feeds into the running windowed aggregation (it becomes part of "the second floor's five-minute average"), it gets checked against any rule that cares about it (is this a spike worth reacting to?), and it carries on down toward storage. Same reading, three jobs, effectively at once.

On the hot path, if it contributed to a live figure or tripped a threshold, a dashboard tile updates or an alert fires — within seconds of the sensor speaking. On the cold path, it lands in cheap durable storage, joining months of its siblings, where one day it'll be a single row in a query asking "what was the second floor's temperature pattern last quarter?"

That's the whole architecture, walked at the scale of one reading: authenticated in, forked in flight, reacted to live and archived for later. And every gnarly thing I've mentioned — the late arrivals, the schema drift, the lying sensors — is some way this tidy journey gets disrupted in reality. The happy path is simple. The platform's real substance is everything that keeps the journey working when the reading arrives late, malformed, or not at all.

What I'd tell someone starting one

If you're about to build one of these, two things. First, split your hot and cold paths from day one — it's painful to retrofit and clarifying to design in. Second, and more importantly: assume the data is dirty and design for that from the start, because the biggest gap between the tutorial and the reality isn't the architecture, it's that the tutorial's sensors behaved and yours won't.

The genuinely satisfying part, the reason I'm enjoying this so much, is that when it works, a building stops being an opaque box and starts telling you about itself — where people actually are, how energy actually flows, what's actually happening inside it. Getting there is less about assembling the impressive components and more about patiently handling all the ways the real world refuses to be tidy. Which, now that I write it down, might be the whole job of data engineering, just with sensors attached.