By: Nicholas Kenney
Introduction
Microsoft Fabric is an end-to-end analytics and data platform that integrates services such as:
- Power BI
- Data Factory
- Data Activator
- Industry Solutions
- Real-Time Intelligence
- Synapse Data Engineering
- Synapse Data Science
- Synapse Data Warehouse
This unification simplifies data management, analytics, and governance, and provides a seamless experience for data professionals. As part of that seamless experience, Microsoft Fabric provides a simplified way to get, process, and store information at real-time.
To learn more about Microsoft Fabric, please see our Microsoft Fabric consulting services page.
Streaming Simplified With Microsoft Fabric Event Streams
Data streaming in Microsoft Fabric is simplified thanks to event streams and event houses. These features make it easy for users to collect, process, and store streaming data.
Event streams is a no-code feature that lets you capture and send real-time events to different places. This means the platform can ingest and process data as it is generated, in real-time, without waiting to accumulate a batch of events (otherwise known as micro-batching).
Streaming data that has been captured and processed needs to go into storage. For that, the data can be put into an event house and then a KQL database. An event house is a solution designed to handle real-time data streams. This capability is unique to Microsoft Fabric and is one of the key ways Fabric makes data streaming easier, in terms of visibility and ease of use.
Once the data is stored, it can be used to make a real-time dashboard (this feature is currently in preview).
Here is a screenshot of data streaming storage solution in Microsoft Fabric:
Benefits of Microsoft Fabric Event Streams
Microsoft Fabric’s event stream is a versatile tool designed for real-time data ingestion, processing, and routing. It can replace or complement technologies like Apache Kafka, Flink, and Cassandra. These are common tools used in a typical real-time data streaming tech stack.
Here’s a table that highlights data streaming with a Microsoft Fabric tech stack versus an Apache tech stack:
From a tech perspective and according to Microsoft, “the Fabric event streams feature offers an Apache Kafka endpoint on the Event stream item, enabling users to connect and consume streaming events through the Kafka protocol.”
Other benefits of streaming in Fabric include its low-code aspect and ease of use.
Example Streaming Pipeline in Microsoft Fabric
1. Data Ingestion
- Azure Event Hubs: Collects and ingests real-time data from various sources, such as IoT devices or application logs.
2. Stream Processing
- Event Stream: Transforms and routes the ingested data. You can apply transformations, filter data, and send them to different destinations.
3. Data Storage
- KQL Database in an Event house: Stores the processed data, providing a scalable and highly available database for querying and further analysis.
Example Streaming Scenario in Microsoft Fabric
1. Ingesting IoT Data
- Azure Event Hubs collects telemetry data from IoT devices.
2. Processing Data
- Event stream processes the data, applying necessary transformations and filtering..
3. Storing Data
- The processed data is stored in a KQL database within an event house, where it can be queried and analyzed in real-time.
Conclusion
Microsoft Fabric provides a comprehensive visualization of the entire data process, from ETL to post-load activities, encompassing various data types and monitoring. While some features are revamped (like Data Factory) others (such as event streams) offer low-code simplicity. Essentially, it encapsulates the analytics department’s purpose into an easy-to-understand template. Its low-code nature means that not everything has a steep learning curve.
