Databricks now frames its Data Intelligence Platform around four pillars: Choice, Control, Cost, and Context. The maturing platform is taking on more work, however, there's still plenty for you to do, too.
Author: Jeremy Marx
Maturity is an easy quality to claim and a hard one to demonstrate.
When it comes to data platforms, there are practical ways to judge maturity. A maturing platform takes on work you used to do yourself. The required business tasks stay the same, however, who is shouldering the responsibility for completing them changes.
At this year’s Data and AI Summit, Databricks organized their platform announcements around four pillars: Choice, Control, Cost, and Context. None of these concepts are new. Anyone who has run data infrastructure has already been solving for all four, usually with some combination of spreadsheets, a wiki nobody reads, and a colleague who remembers how that one esoteric metric is actually defined. What changed this year is how much of that weight the platform is willing to carry.
Earlier this year we looked at how Databricks built a Data Intelligence Platform: a governed foundation in Unity Catalog, natural language access to data through Genie, and agents grounded in an organization’s own information. That foundation has not changed. What’s new is the discipline being built on top of it, and the shape of the work left over for you.
Databricks’ four pillars of Choice, Control, Cost, and Context are not a product list. They are the properties a system needs before it can be trusted to act on a real business decision. Product-style capabilities sit behind each one; however, the pillar is the durable part. Products will keep changing. The tasks that need to be completed and the quality expected from a system will not.
These pillars are also not independent, which turns out to be the most practical thing Databricks engineered into their updates.
Choice means your data, your models, and your workloads are not held hostage by a single vendor’s runtime. Databricks’ foundation already worked this way, with Unity Catalog governing tables, files, notebooks, models, dashboards, and agents across workloads and clouds, and open formats underneath.
This year, they extended the principle past data. OpenSharing shares not just tables but models, agents, skills, and semantic context, with recipients querying live data instead of receiving copies. It is also a vendor-neutral open-source project rather than a proprietary Databricks feature, with a published specification anyone is free to implement. This is an entirely different level of commitment to open source than most vendors offer.
The AI model layer widened too. Grok arrived through a partnership with xAI, and Databricks added Kimi, a cheaper and more token-efficient option, at the low end of the range, joining Claude, GPT, Gemini, and Qwen. Omnigent †, which Databricks released as open source, sits above the agent harnesses your developers already use. It lets them combine models and harnesses, fork a running session from one into another, hand a live session to a colleague with its full context intact, and apply policies and spending limits across all of it. It also runs perfectly well without Databricks, which is an unusual thing for a vendor to build on purpose.
The pillar of ‘Choice’ asks one thing of you: foresight. Nobody plans for lock-in; it happens in the background, while you are simply trying to make things work within the constraints you already have.
Choice—portability—deserves to be a design-time consideration rather than a later discovery. The useful question is not whether you could move, but which pieces are already immobile.
Governance used to mean deciding who could see which rows. Now, that definition is far too simple for the reality of AI.
Agents not only read your data, but act upon it in live systems based on their findings. That is the entire appeal and the entire problem.
Databricks extended their foundational product rather than replacing it. Unity Catalog now registers agents, tools, and MCP servers as governed assets alongside tables, dashboards, and models. This way, the catalog that answers “who can see this” also answers “what is this allowed to do.” Contextual policies † judge the live circumstances of an action rather than consulting a static list. Databricks gives a clarifying example: an agent that has just read personal data might be allowed to email a colleague, but not to publish to a public site, and might need human approval before writing to a system of record.
Lake Watch† takes the same idea somewhere less obvious. Security tooling has long been priced in a way that punishes you for keeping data, so teams filter their logs down to what they can afford to retain. Nobody enjoys deciding which evidence to throw away, and that trade looks worse now that attackers have agents of their own and alert volumes run into the tens of thousands per week. Lake Watch keeps the whole picture in a governed data lake, where storage is cheap and open, and puts agents on the manual triage work that was burying the people doing it.
What ‘Control’ requires is an inventory and an owner. Every agent touching your data needs a stated purpose, a boundary, and a person accountable for both. This seems trivial when you have three agents, however, when teams have three hundred, not having it becomes a liability.
Unity Catalog will keep a list of agents now, which is real progress. A human still needs to decide what each of them ought to be allowed to do.
Cost discipline is not new to Databricks either. Predictive Optimization and Liquid Clustering were already trimming storage and compute by learning from real usage, and Agent Bricks already tuned agents for cost as well as quality. In the meantime, what has changed is how that spend is calculated.
Token usage is harder to forecast than a storage bill, which grows more or less in step with your data. An agent that cannot find an answer doesn’t get discouraged and cease working. It tries again, a little differently, and again after that. That persistence is key to an agent’s utility, and it is also how even a narrow, well-defined question becomes an expensive one.
Anyone who has caught a query still running an hour after it should have finished knows how quietly compute costs pile up. An agent makes the same mess in a different shape: not one long query you eventually notice, but a rapid series of short attempts, each one perfectly defensible on its own. Multiply that across an organization and the numbers get serious. Databricks remarked during the keynote that the top one percent of AI spenders were running at roughly $90,000 per user per year.
Unity AI Gateway is the answer to that. It gives you one entry point for agents, models, and MCP servers, and it makes spend legible rather than merely large. Every request is logged to Unity Catalog with DBU costs and also token counts. Budgets can be set for the organization and per person. Crossing one raises an alert by default, and if you configure a hard cap, the Gateway stops further requests until someone raises the limit or the next billing period begins.
Perhaps the most interesting piece is model routing. Instead of sending every request to the most capable model available, Unity AI Gateway can route by complexity, letting an inexpensive model handle ordinary work and reserving the expensive one for problems that earn it. Databricks put it on stage as moving from “token maxing” to “value maxing,” which is both accurate and hard to forget.
‘Cost’ demands that you set a spending policy before you need one. Which teams own which spending, where the thresholds sit, and what happens when someone crosses one are all easier conversations to have now than after an invoice arrives with a question mark attached.
The first three pillars dictate how a system behaves. Context decides whether it is right.
In March we wrote that Data Intelligence is built in, but not self-sufficient. The platform ships with real intelligence, yet it still does not know that “active customer” means something specific in your business, or which of your revenue tables people actually trust. Absent that, an agent fills the gap with a guess, which Databricks describes, fairly, as generic at best and wrong at worst.
This year’s answer to the context problem was a reorganization more than a single product. Genie One is the organization-wide assistant for asking questions of data. Genie Agents † turn a useful conversation into a repeatable agent you can schedule and share. Genie Code writes and maintains data engineering and machine learning work. Genie ZeroOps, available in private preview as of this writing, runs quietly in the background, watching pipelines and models, diagnosing what broke, and bringing a proposed fix for a human to approve. Agent Bricks remains where you build agents specific to your organization.
What ties them together sits underneath. Genie Ontology, not yet available to all customers, assembles a living map of what your data means, drawn from tables, queries, dashboards, and pipelines, and from the places business context actually accumulates, including documents, ticketing systems, and SharePoint. When two definitions of a metric disagree, and they will, it weighs them by the authority of the source, how heavily the asset is used and certified, and how fresh the definition is. Source permissions are enforced throughout, so context can be assembled without accidentally leaking anything.
The effect is measurable. Databricks reports that on a 28-question benchmark of real business questions, Genie grounded in the ontology answered about 85% correctly, against 52% for the strongest general-purpose coding agent it tested.
What ‘Context’ needs has shifted almost imperceptibly, and that is the part worth sitting with. Ontology ranks what it finds, and it cannot rank what nobody wrote down. If your most-used dashboard encodes a definition no one is willing to defend, that is the definition it will learn.
The real work is curation and sourcing, and both carry more weight now than they used to. Curation means the definitions that matter are written down, agreed upon, and certified, so that when the ontology weighs competing versions, the one you would stand behind is the one that wins. Sourcing means that context has to live somewhere the platform can actually read. A definition held only in a senior analyst’s head, or in a chat thread from two years ago, is invisible to an ontology no matter how good the ranking is.
Curation is an input to whether the answers are right, and it compounds: every definition you settle makes the next question easier to get correct.
These four pillars are usually presented as separate lanes. In practice they interact, and an interaction worth understanding sits between Context and Cost.
Think of Genie as billing through two meters rather than one. The first is language model usage, which the AI Gateway governs and documents. The second is the compute that runs the queries the model generates.
Now picture how a thin layer of context complicates this. A question arrives with no governed definition behind it, so the agent guesses, writes something broad, scans more than it needed to, gets an unsatisfying result, and tries a different join. Every attempt is compute. That spend lands on the second, hidden meter that your AI budget is not watching, and no cap will catch it, because from the budget’s point of view nothing has gone wrong.
Better context shortens that loop. A well-defined metric turns an exploratory scan into a narrow, correct query on the first attempt. The savings arrive somewhere other than where the discipline was applied, which is a large part of why nobody goes looking for them.
That is the general shape of it. Rigor in one pillar tends to pay off in another. Treated as a checklist, the four pillars are simply four chores. Treated as a system, they start doing some of the work for you.
Whether you have agents in production today or are still deciding where to start, the same four questions apply. These are the types of questions worth serious contemplation, and they are deliberately harder than they look.
1) If you needed to run a workload on a different engine, or give a partner access to your data without sending them a copy, what would break?
2) Can you name every agent with access to your data, and say what each one is allowed to do? How would you answer that a year from now, with far more of them?
3) Do you know which AI work costs the most per useful answer, and which the least, rather than in total? And if consumption started growing faster than the value it returned, what in your setup would stop it?
4) If two teams asked the platform the same business question today, would they get the same number, and could you show why?
Few organizations can answer all four with confidence. The pillars are old, but this is the first year the platform has been able to carry much of the weight, so for many, this is the first time these questions will be asked.
It is worth turning the same critical eye on your own data practice. If a maturing platform is one that takes on more of the work, then a maturing data organization is one that can answer more of these questions without hedging.
Maturity never reaches a final, perfect state. This should be seen as encouraging. Nobody is behind. Every definition settled, every agent given an owner, and every threshold agreed on moves one answer from uncertain to solid, and the returns compound from there.
Finding those answers, deciding which step comes next, and taking it with confidence is the work. It is also where we come in. Contact us today to chat with one of our Databricks experts about where the platform can find value in your organization.
† Several capabilities described here were announced recently and may not be available yet in every region or to every customer. Check current availability before planning around any of them.
Jeremy Marx is a Staff Engineer at RevGen, specializing in data engineering. He has over a decade of experience building and managing data systems and pipelines on multiple platforms, dedicated to ensuring data consistency that drives meaningful business insights.
We recommend the same AI workforce enablement framework for our clients that we use ourselves because we know it works.
Read More
These 4 change management techniques are key to ensuring you get the ROI you expect from your AI investments.
Read More
RevGeneration closes the revenue intelligence gap by finding exactly where your revenue opportunities hide and implementing solutions that deliver real results.
Read MoreGet the latest updates and Insights from RevGen delivered straight to your inbox.