Author: David Thomas
With all the buzz about AI generation and prediction and LLMs and cyber-this and intelligent-that it can be hard to make sense of what tools are helpful and what help is effective. So much of what AI tools are generating seems to be amazing one minute and ridiculous the next. For organizational decision makers, struggling to make sense of overwhelming and conflicting information is nothing new. It can be incredibly difficult to sort through what information has value and what that valuable information tells you and your team about the future. Traditionally, sifting through all that data has been the domain of computer server rooms, highly paid analysts, and technical experts. For decades, that has left us dependent on teams of people and expensive equipment to attempt to quantify and summarize, aggregate, and estimate while our decisions continue to depend heavily on instinct and intuition.
Enter the AI hype train. Now we have companies replacing humans with poorly defined, AI-built tools that are generated quickly but present glaring failures almost as quickly. The motive behind attempting to use AI as a blunt instrument is understandable. The cost of getting accurate, reliable, actionable intelligence has been prohibitive, especially for small and mid-sized organizations that do not have the resources to invest in larger initiatives. Naturally, we don’t want some AI agent built by someone else to make mission critical decisions for us, but we need quick turnaround on pivotal questions in high pressure situations. We don’t want to sit through two months of meetings to get a guess. We cannot wait six months to onboard talent that costs more than we can afford. We don’t want to read yet another proposal for an expensive service or more expensive equipment. In short, we want the walls between us and our data to be lowered!
Enter Databricks free edition. This is not a free trial that ends in five days and uses most of them for tutorials. This is not a free app that lets you type a couple of sentences to generate a chart that is based on conjecture and guesses. This is a powerful set of tools made available to you so that you can plug in some of your data and start asking some questions. This is a sandbox with an AI assistant to help walk you through creating a dashboard to answer key questions in a short amount of time. This is the great paywall being lowered to give you access to actionable intelligence. Is it going to give you everything you want, need, or ever hoped for at the press of a magic button? No, of course not. There is no magic button to be pressed that bypasses reality or reason. It is, however, going to get you started exploring your data in something other than an outdated, inaccurate, printed spreadsheet that took far too much time and money to produce.
I created a Databricks free edition account the other day. It only took a few minutes and what I got was a remarkable sandbox of tools to play with. The first thing I asked was, of course, how do I get my data into this thing? Specifically, I wanted to set it up for a specific non-profit organization so that I could start analyzing financial trends. To keep the setup simple, I started with some old favorites; a spreadsheet of contacts and a spreadsheet of donations. Yay spreadsheets! Lo and behold, the Databricks team anticipated this and with a click on Catalog and then a click on the plus button on the panel that opened, I was given the ‘Add Data’ option. From there, I could have chosen to add my files to a volume (a storage location), but I decided to try Fivetran’s ‘Google Drive’ option because my spreadsheets were already in one of my Google Drive folders. Fivetran gave me a 14-day free trial, so I thought I would see how simple they could make it. What I got was my spreadsheets automagically loaded from my Google Drive folder into tables in my Databricks catalog. They may have been a little ugly, but I did not spend any time making them pretty and that was acceptable for the test. I was able to begin exploring my data and experimenting with it within 30 minutes of starting!
After spending a little time curiously poking around, I refocused and headed over to Dashboards. I clicked the option for creating a new dashboard and Databricks walked me through selecting the data I had imported from Google Drive and adding my first visualization. Using the AI assistant, I was able to use natural language to define what I wanted to see and produce some interesting and helpful visuals in minutes. The real magic, however, came when I scrolled through the settings for one of the visuals. Down at the bottom, of the settings for my ‘Annual Donation Trend’ chart, there was a button for Forecast that was in Beta testing. I clicked it and gave it permission to ‘Clone with AI Forecast’. After a couple of minutes, I had a new chart that provided a prediction about donations year over year that was based on an analysis of the trend my imported data showed!
So, why then, is a data professional celebrating such an easy to use, free tool being made available to decision makers that would otherwise be paying HIM to import this data and build these charts? The answer is simple; I’m in this business to help you get actionable intelligence. I generated and refined these charts with a couple of hours of work at most. If I imported all the data, cleaned it up properly, refined it into easier-to-use structures, and built dashboards targeted at answering specific questions, I could get so much more out of my data. It does take time to find the questions, the right data for answering each question, and put everything together.
With Databricks free edition, my organization was able to get started without spending one precious dime. With some help from automations, I was able to skip hours of tedium and start exploring some data so I could better see what questions I wanted to ask
and what additional data was needed. With the experimental forecasting option, I was even able to get a quick glance at what our recent financial trends might be saying about our near future. I was able to start the journey quickly and well within budget! If I add to that other free and low-cost storage options, I can continue to build from there to collect the data I need and answer the questions I have.
The reality is, our data, just as in every organization, is not perfect. It is dirty, duplicative, and disorganized even on a good day. As frustrating as that can be at times, it is a natural byproduct of attempting to bring order to the chaos that results from operational plans meeting operational reality. Having the wall lowered to getting actionable intelligence does not magically make all those problems go away; it gives you a starting point that does not require yet another financial investment. It helps us start a conversation about what information you have and what answers you need. It helps us see through each other’s eyes so that we can communicate more effectively, have fewer meetings about data, and produce accurate answers faster than ever before without having to spend years trying to get started. It helps us operate more effectively together, and that is priceless.
More reading and resources:
Databricks Consulting Services | Key2 Consulting
Databricks: Leading Data and AI Solutions for Enterprises
Free Edition | Replacing Databricks Community Edition
Free Trial and Free Tier Services and Products | Google Cloud
