What is Azure Databricks?

Databricks is a unified cloud analytics platform built for working with Apache Spark. In fact, the creators of Apache Spark are the same people who created Databricks. The Databricks platform has a Microsoft platform integration called Azure Databricks, which was announced in the fall of 2017.
 

What are the key features of Azure Databricks?

Azure Databricks’s general purpose is to help organizations simplify big data. The cloud platform offers users two options within the workspace: Data Science & Engineering and Machine Learning.
 

Data Science & Engineering

The Data Science & Engineering workspace option is best used for data ingestion, data engineering, and data science work efforts. Note that a Databricks workspace can be used within an Azure Data Factory pipeline during data ingestion and/or data transformation.
 

Machine Learning

The Machine Learning workspace option is best used for machine learning/artificial intelligence work efforts. Databricks offers CPU and GPU-based clusters through its runtime as part of both its data science and machine learning offerings.

The clusters typically come preinstalled with most current python and r machine learning libraries such as tensorflow, pytorch, etc. Databricks also offers specific library/version installs through its Compute/Cluster UI from PyPI, CRAN, or Maven.

The Machine Learning workspace supports the model development lifecycle, from developing an ML model (including training & testing) to deploying the model to updating the model in Databricks.

The cloud platform has an Experiments feature that can be leveraged by a team of ML engineers to run several experiments. Azure Databricks registers the several experiments run by users using the ML Flow feature. This feature includes timestamp, name of an experiment, user, and result metrics in an easy-to-view dashboard. Models can be registered in staging and further in production through the Models feature in Databricks.

Another great feature of the Machine Learning option within Databricks is the available selection of cluster categories. Under each category, there is usually several machine type options based on number of CPUs and RAM. The different cluster categories are:

• general purpose virtual machines
• memory optimized virtual machines
• storage optimized virtual machines
• compute optimized virtual machines
• GPU accelerated virtual machines

Users have the flexibility to choose virtual machines from any one of these categories based on their work effort. Clusters can be autoscaled from one virtual machine to several depending on the code or query complexity.

Clusters can also be auto terminated after a few minutes of idle time. Both the autoscaling and auto termination features provide cost savings to the end user since Databricks charges on cluster size and cluster usage time.
 

Notebook-Style Coding

Databricks offers notebook-style coding. Code can be written in four languages – Python, R, SQL (hive and spark), Scala. Code written in notebooks can be attached and run on a cluster.

Tables and/or data in various formats can be directly read into Databricks notebooks from Azure Gen2 Data Lake Storage through a Service Principal client or AAD Passthrough enabled on clusters. Transformed data can also be conveniently written back to Azure Gen2 Data Lake Storage or hosted within the Databricks workspace as hive tables.

Databricks supports version control on notebooks with GitHub, Bitbucket Cloud, or Azure DevOps integration. Users can integrate notebooks with their group/team repository in GitHub or Bitbucket or Azure DevOps and check their code as needed.
 

Automated Jobs

Databricks also offers a Jobs feature to kickoff automated jobs on a scheduled basis. The jobs feature is ideal for data engineering and/or data transformation work efforts for ELT jobs, and supports the following:
 

• scheduling a notebook or series of notebooks to kickoff at a certain day/time
• scheduling a JAR
• scheduling a Python file
• spark-submit jobs

Who is Azure Databricks designed for?

Databricks is designed for a variety of data practitioners: analysts, data engineers, data scientists, and so on. It is a platform designed to bring data science and business together to boost innovation and empower users to make better use of their data.
 

Our Latest Azure Content