A Quick Look at Azure Databricks Secret Scopes

November 29, 2021
Azure Databricks secret scopes is an excellent tool for creating effective data security measures and protecting sensitive data.

By: Phillip Sharpless

 
When building solutions with Azure Databricks (see my previous blog post on Databricks for a quick overview of the analytics service), it’s more than likely that the need to access secure data will arise.

Access to some data may require sensitive information, such as passwords, access tokens, or account names. These are things you really should not have hardcoded within an application or have sitting around in a file somewhere.

How you handle securely accessing and using credentials within any application is always a major security concern. Azure Databricks has an excellent construct for such situations known as secret scopes.

A secret scope serves as secure storage for sensitive information. It also represents a logical container, such that sensitive information can be logically grouped and organized within different scopes for different audiences. Secret scopes store secrets, which are key-value pairs that contain the sensitive information.

Secret scopes can be managed in one of two ways:

  • they can be stored directly within an encrypted database managed by Azure Databricks
  • they can be managed by the Azure Key Vault

Secret scope permissions are managed by something known as Access Control Lists (ACLs). Different people or groups can be assigned to only the scopes they need. There are also varying levels of access permission.

Manage, the highest level of access, gives users the ability to fully read from and write to the scope as well as manage the ACLs associated with it. Write gives users the ability to read from and write to the scope but no ACL access. And finally, Read gives users the ability to read from the scope with no write access.

In summary, secret scopes is an excellent tool for securing and storing sensitive information in Azure Databricks and/or Azure Key Vault. If you’re looking to enhance your data security and are currently working with either of these Azure tools, we highly recommend secret scopes.
 
 

Thanks for Reading! Questions?

Thanks for reading! We hope you found this blog post useful. Feel free to let us know if you have any questions about this article by simply leaving a comment below. We will reply as quickly as we can.
 
 

Keep Your Data Analytics Knowledge Sharp

Get fresh Key2 content and more delivered right to your inbox!
 

 
 

About Us

Key2 Consulting is a boutique data analytics consultancy that helps business leaders make better business decisions. We are a Microsoft Gold-Certified Partner and are located in Atlanta, Georgia. Learn more here.

Related Content
How to Test Azure Data Factory Linked Services Using PowerShell

How to Test Azure Data Factory Linked Services Using PowerShell

By: Brad Harris       The Problem Over the past couple of months, I have been working with Azure Data Factory (ADF) in a project where we are migrating all of our on-prem data to the cloud using a combination of Azure Data Factory and Azure Databricks....

What is Azure Cosmos DB? Five Key Features

What is Azure Cosmos DB? Five Key Features

By: Brad Lathrop   Are you exploring database options? Considering moving data to the cloud? Work for a company that utilizes a plethora of Microsoft products? If you answered yes to any of those questions, chances are you may have come across Azure Cosmos DB at...

4 Key Features of Databricks Machine Learning

4 Key Features of Databricks Machine Learning

By: Brad Lathrop   Azure Databricks is a Microsoft data analytics platform that’s “optimized for the Microsoft Azure cloud services platform” (1). It has three environments developers can choose from to create applications: Databricks Machine Learning Databricks...