By: Matt Wisor
In early November, Microsoft announced the release of Azure Synapse Analytics at the Ignite Conference in Orlando. The new analytics service is intended to help businesses bring data warehousing and big data together for deeper, more comprehensive analysis.
Synapse Analytics is the next generation of SQL Data Warehouse, re-engineered to combine data warehousing and big data analytics into one service platform.
An Innovative Step in Cloud Analytics
While enterprise data warehousing and big data solutions are common, many organizations lack the expertise and staff to hook them together for analysis. Microsoft is breaking down these barriers with its Open Data Initiative (ODI) and by collaborating with partners such as Databricks, Informatica, Attunity, and Panoply.
Synapse Analytics enables businesses to consolidate, maintain, and perform analysis on data from many disparate systems. To date, Synapse Analytics is able to consume data from over 85 native connectors.
How It Works
Data is ingested from various sources and stored in an Azure Data Lake. Using Hadoop, Spark, and machine learning processes, data is then cleansed and trained. Once the data has been prepared, it is ready to be loaded into a relational SQL Analytics data warehouse using PolyBase. The SQL Analytics warehouse stores data in a columnar format for optimal query performance and space savings.
The SQL Data Warehouse engine was completely redesigned for Synapse Analytics. The enhanced engine uses a Massively Parallel Processing (MPP) architecture to process queries at lightning speed. An intelligent distributed query optimizer submits queries to a control node which divides the workload across multiple compute nodes for processing in parallel. As a result, analytics queries complete in a fraction of the time when compared to traditional data warehouses.
Rohan Kumar, Vice President of Azure Data Engineering, gave a product demonstration at the Ignite conference and said, “It’s the first and only analytics system to have run all TPC-H queries at petabyte-scale.” The TPC-H benchmark is a standard industry benchmark used to measure data warehouse system performance.
Key New Features
Here are some new features beyond what was generally available in SQL Data Warehouse. A more complete list of all features can be found in Microsoft’s documentation.
- The Azure Synapse Studio is a unified workspace for all data professionals using the Synapse Analytics service. Data scientists, data engineers, DBAs, and business analysts are all able to carry out tasks on both relational and unstructured data using standard SQL.
- The PREDICT function is now available for analysts to generate a predicted value or scores based on a model (in preview).
- Direct streaming ingestion offers real time data streaming and the ability to run queries over the live stream (in preview).
- ParquetDirect technology uses an integrated Spark engine built into Synapse, not Databricks, to query the data lake directly. It natively interacts with files stored in the Apache Parquet columnar format (in preview).
- Workload isolation provides the ability to define workload groups and reserve resources for priority workloads (in preview).
- SQL Server Data Tools (SSDT) Support for continuous integration and deployment (CI/CD).
Security
Data security is one of the top priorities for businesses operating in the cloud. Knowing this, Microsoft has built cutting edge security and protections into its Synapse Analytics service. Automated threat detection and always-on-data-encryption are at the forefront, while column level security and row level security provide more targeted data access. Lastly, dynamic data masking protects sensitive data in real time automatically.
Do you need help with Azure?
If you have a need, please feel free to contact us at anytime. We can help you design and develop a new Azure solution or customize/adapt an existing one.
Questions?
Thanks for reading. We hope you found this blog post to be useful. Do let us know if you have any questions or topic ideas related to BI, analytics, the cloud, machine learning, SQL Server, (Star Wars), or anything else of the like that you’d like us to write about. Simply leave us a comment below, and we’ll see what we can do!
Keep Your Business Intelligence Knowledge Sharp by Subscribing to our Email List
Get fresh Key2 content around Business Intelligence, Data Warehousing, Analytics, and more delivered right to your inbox!
Key2 Consulting is a data warehousing and business intelligence company located in Atlanta, Georgia. We create and deliver custom data warehouse solutions, business intelligence solutions, and custom applications.