Orchestration Optimization Opportunities Guide
The Orchestration tab in Executive Insights answers a question that is hard to answer anywhere else: what is your Databricks estate connected to, and what is that costing you?
Every opportunity in this category is the same shape. Something is running in two places when it could run in one — two platforms, two storage formats, two metastores, two orchestrators. Orchestration finds those duplications and shows you what it would take to close them.
Unlike cost and performance findings, these are structural. They tend to be larger pieces of work with longer payback, and they are the ones most likely to need a decision rather than a configuration change.
💼 How to read this tab as a business. None of these findings says turn something off. Resources that drive business outcomes should be running — the question this tab asks is whether they are running in the right place. Every card is a form of the same problem: paying twice for something the platform already does. That makes Orchestration an architecture conversation rather than a cost-cutting one, and it is why these opportunities tend to belong on a platform-strategy agenda rather than a monthly cost review.
One thing to be clear about up front. This tab produces no savings estimate. The cost shown against each opportunity is what you are spending today on the workloads involved — your Databricks cost, measured from your own telemetry. It is a way to rank what to look at first, not a projection of what you would save. What consolidation is worth in your estate depends on decisions the tab cannot see.
What the tab shows
Four opportunity types, each as a tile:
Opportunity | What it counts |
|---|---|
Duplicated vendor TCO | Workloads reading from or writing to an external data platform |
Inefficient cloud storage access | Workloads reading raw files from cloud storage |
Unity Catalog migration incomplete | Workloads still using the Hive metastore or DBFS |
External orchestrators | Workloads triggered from an orchestrator outside Databricks |
Each tile carries the number of workloads affected, how that has moved since the previous period, and a complexity indicator — a rough guide to how much work the change involves, not a measure of its value. Tiles are ordered by how many workloads each one affects, so the arrangement reflects your estate rather than a fixed priority.
Selecting a tile filters the workspace list beneath it, so you can see which parts of your estate the opportunity sits in.
Why this is hard to see elsewhere
Inside the Databricks console you can see what runs in Databricks. What you cannot easily see is what Databricks is connected to.
External orchestrators appear as one-time runs, without the context of what created them or how large that footprint is. Vendor connections and cloud-storage reads live inside job code, so finding them means reading the code — which does not scale past a handful of jobs. And while a Hive metastore lists its own tables, what it cannot tell you is which of them your jobs are still using — that mapping only exists in the code as well.
LHO reads all of this from execution telemetry instead, which is why the picture is available without anyone auditing code.
#1 Duplicated vendor TCO
What it is
Workloads that read from or write to another data platform — Snowflake, Teradata, Oracle, SQL Server, Redshift and similar — directly from Spark. The direction matters: your Databricks workloads reach out to those systems, so what you see here is measured entirely from the Databricks side.
What we detect
LHO reads the Spark execution plan for every monitored job, so it sees which external systems your Databricks workloads actually connect to. No code changes or annotations are needed — the workspace simply has to be monitored by LHO.
Why it matters
Running two data platforms in parallel means paying for the same capability in two places — storage, compute, and the engineering time to maintain each of them. Teams need expertise in both, and pipelines get built twice.
On top of that, every dataset that moves between the two platforms carries a network transfer cost and adds latency, and cross-region movement compounds both.
Not every finding is worth acting on. Transactional systems such as Oracle or PostgreSQL often belong where they are. The value of the view is that it tells you which vendors your Databricks workloads actually touch, and how much those workloads cost — which is usually different from what people assume.
💼 The business case. Total cost of ownership here is broader than two invoices. It is duplicated storage, duplicated compute, and — the part that rarely makes the spreadsheet — duplicated people: two platforms mean two sets of expertise on the payroll and every pipeline built twice. There is also a cost almost nobody measures: while a Databricks workload waits on the other system, your cluster is running and being billed for the wait. This tile is where a consolidation conversation stops being run on opinion — instead of a general sense that too much is spent moving data around, you have a ranked list of the workloads tethered to an outside system and what each one costs to run.
Where to go next
First, stay on the page. Selecting the Duplicated vendor TCO tile filters the workspace list beneath it — and each workspace then shows which platforms specifically it depends on, named, with the number of workloads involved and whether that has grown or shrunk since the previous period.
That is the step most people miss, and it is the one that turns a single estate-wide number into something you can act on. One workspace may show Snowflake; another MS SQL Server; a third both. Before you go anywhere, you already know where each dependency lives and which direction it is moving.
Then, for the workloads themselves: Vendor Consolidation → Migration Opportunities.
Vendor Consolidation is its own item in the left navigation, under Optimize. It opens on Transferred Data & Cost, which is the overview. Migration Opportunities is the second tab — that is the one you want, because it ranks individual workloads rather than summarising the footprint.
Set the same month, then filter to the workspace and the platform you identified above.
What you get there: the workloads associated with that platform, ranked by cost — so you can see whether the dependency is concentrated in a handful of pipelines, which is usually the case and usually the thing that decides whether a migration is worth starting. The list shows the highest-cost workloads rather than all of them, so expect it to be shorter than the count on the tile.
How to read the cost column: when one workload reaches two different platforms, its full cost is attributed to each of them. That is deliberate — it keeps each vendor's picture complete — but it means the column ranks workloads rather than summing to a total. Sort by it; don't add it up.
#2 Inefficient cloud storage access
What it is
Workloads reading raw files straight from cloud storage — ADLS, S3, Azure Blob, GCS or HDFS — in formats like Parquet, CSV or JSON, instead of reading Delta tables.
What we detect
The storage paths your monitored jobs actually read, taken from the Spark execution plan — so the count reflects live usage rather than everything present in the account.
Why it matters
Listing a large number of small files through cloud APIs is slow and expensive, and that metadata cost grows with the number of files rather than the amount of data. Delta tables avoid it: reads are faster and cheaper, and the tables can be optimized and vacuumed on a schedule.
Reading across clouds adds egress charges on every run, which makes the same pattern more expensive again.
There is a second reason to look at this list. A large cloud-storage footprint often points to a system you have not accounted for. Those files came from somewhere. When two platforms cannot talk to each other directly, they usually share data through cloud storage instead — so a set of unexplained reads can be the only visible trace of a system that is otherwise invisible to your Databricks monitoring.
One thing that looks like an error and is not. A workload showing cost but no external storage at all is the healthy case — it read only Delta tables through Unity Catalog.
One expected exception: ingestion has to start somewhere. A bronze or ingestion job reading source files from cloud storage is doing exactly what it should. What matters is what happens downstream — jobs further along the pipeline should be reading Delta tables, not going back to the raw files.
What to do: convert the datasets these workloads read into optimized Delta tables, and schedule optimization and vacuuming.
💼 The business case. This is the recurring-cost pattern on the tab: the listing overhead and any egress are paid on every run, and a one-time conversion removes them from every future run — which makes the payback arithmetic unusually clean for work of this size. The second half of the case is discovery: a large unexplained storage footprint is often the only visible trace of another system sharing data with Databricks through storage, which means this tile can surface a platform dependency — and a TCO conversation — that no inventory would have found.
Where to go next
Same path as the vendor opportunity, filtered to a storage type instead of a platform.
Cloud storage appears in Vendor Consolidation alongside the external platforms — AWS S3, Azure Data Lake Storage, Azure Blob Storage, Google Cloud Storage and HDFS are all filterable there in exactly the same way.
Stay on the Orchestration tab first. Selecting the tile filters the workspace list, and each workspace shows which storage types it reads from and how many workloads are involved. Start where the footprint is.
Then go to Vendor Consolidation → Migration Opportunities — left navigation, under Optimize, second tab — and filter to the storage type and month you identified.
Open a workload, then a run, then its tasks to reach the storage path itself.
What you get at the end: the actual path — container or bucket, folder structure and file format. That is what separates a real problem from an ingestion job doing its job. Thousands of small date-partitioned JSON files are the first; a single well-formed source drop is the second. You cannot tell which you have from the count alone, which is why this last step matters more here than anywhere else on the tab.
#3 Unity Catalog migration incomplete
What it is
Workloads still reading from the Hive metastore or from DBFS paths rather than from Unity Catalog. DBFS is the legacy Databricks file system; Hive is the legacy metastore. Both predate Unity Catalog, and most long-running Databricks estates still have some of each.
What we detect
The active footprint — the Hive tables and DBFS paths your jobs are genuinely using, observed continuously as those jobs run. That is a deliberately narrower and more useful number than a full inventory of everything in the metastore, because it tells you what migrating would actually unblock.
Why it matters
Governance first. Data outside Unity Catalog is governed by rules held elsewhere, often by another team. Access can be changed without your knowledge, and jobs depending on it fail. Bringing the data under Unity Catalog puts those permissions in one place.
And governance gates the rest of the platform. Unity Catalog is a prerequisite for Databricks Genie and for a number of the newer Databricks capabilities. An incomplete migration quietly keeps them out of reach.
And duplication. Unity Catalog allows data to be shared across workspaces, so teams no longer need to keep their own copies of the same datasets.
Migration is rarely small — large estates can carry hundreds of thousands of tables — which is exactly why it helps to know which ones are actually in use.
💼 The business case. An incomplete migration carries two costs that compound quietly. The first is risk: data governed outside Unity Catalog is governed by someone else's rules, and a permission change made elsewhere can take your production jobs down without warning. The second is opportunity: a number of the newer Databricks capabilities — Genie among them — require Unity Catalog, so every quarter the migration stays unfinished is a quarter of platform roadmap you are paying for and cannot use. Finishing it is less a tidiness project than the entry ticket to running Databricks as a centre of excellence rather than a collection of workspaces.
Where to go next
This tile counts two things — Hive tables and DBFS paths — and today they lead to different places.
Start on the Orchestration tab. Selecting the tile filters the workspace list to the workspaces that still carry an active footprint, and shows how much each one holds.
What you get there: the remaining migration, distributed. Rather than one number for the estate, you see which workspaces hold the active usage — which is what lets you sequence the work, and which is usually uneven enough to change the plan.
For the DBFS half, go to Vendor Consolidation → Migration Opportunities — left navigation, under Optimize, second tab. DBFS is listed there as Databricks File System and filters exactly like any other source. From a ranked workload you can open its runs, then its tasks, and reach the literal path.
What you get at the end: the specific mounts and paths still in use, and the workloads using them — which is the working list for that half of the migration.
For the Hive half, the workspace distribution above is as far as the product takes you today. The table names behind the count are detected but not yet listed anywhere you can open.
One turn worth not taking. LHO also has a Unity Catalog Migration assessment in the left navigation, and it is not the follow-on to this tile. It answers a different question: it inventories what exists in the metastore, as of the last time you ran it. This tile shows what is still being used, observed continuously as your jobs run. A table in the assessment may be long dormant; a table behind this tile ran this month. Use the assessment to scope the migration as a whole — use this tile to decide what to move first.
#4 External orchestrators
What it is
Databricks work driven by a scheduler that sits outside Databricks, rather than by the orchestrator built into the platform.
This is common, and almost always historical. Databricks did not originally have the breadth of source connectors it has now, so teams built their ingestion in the tool that did and handed the results over. Which tool that was tends to follow the cloud: Azure Data Factory on Azure, where it is very common, and Apache Airflow or a similar scheduler on AWS, where the pattern is rarer but does occur.
The shape is the same whichever tool is involved — a pipeline defined in one system, doing work in another.
What we detect
Today, Azure Data Factory. Jobs that arrive in Databricks as one-time runs created by ADF, identified from the run type and the job naming ADF applies.
Other external schedulers — Airflow, Oozie and similar — are not yet identified, so work triggered by one of those will not appear in this count. The pattern they create is the same one described here, and the detection is designed to extend; if you orchestrate Databricks from something other than ADF, tell us and we can look at covering it.
One consequence worth knowing even on Azure: because detection keys on how ADF names the jobs it creates, a pipeline producing differently named jobs may not be counted. If the number looks lower than you expect, that is the first thing to check.
Why it matters
Three costs travel with this pattern. The first two apply to any external orchestrator; the third is what we see specifically with ADF.
An extra hop. Externally orchestrated pipelines typically land intermediate datasets in cloud storage before Databricks picks them up — a step that exists only because the two systems are separate.
Duplicated expertise. Pipelines are built and maintained in the external tool, and the corresponding work exists on the Databricks side too. That is two skill sets and two sets of code for one pipeline.
Compute that was never chosen. The runs ADF creates tend to execute on all-purpose compute rather than job compute — a higher rate, with idle time between runs, and frequently more capacity than the work needs.
What to do: the reason this pattern exists has largely gone away. Databricks now covers most of these sources natively through Lakeflow Connect, and its own orchestrator can run the pipeline end to end. Consolidating removes the hop, the duplication and — where ADF is involved — the compute mismatch, together.
💼 The business case. The largest line item here is usually people, not compute: an external orchestrator means a second tool to staff, maintain and keep expertise in, for pipelines the platform can now run end to end. And unlike the other three opportunities, this one is unusually clear-cut — the pattern exists for a historical reason that no longer applies, and the destination is known. The open question is usually when, not whether — which makes it a natural early candidate when sequencing the work on this tab.
Where to go next
Optimization Review → Jobs, with the One-Time Runs filter applied.
Optimization Review is a top-level item in the left navigation, and it now covers three kinds of compute — Jobs, All-Purpose Compute and SQL Warehouses. Externally orchestrated work arrives as jobs, so Jobs is the tab you want; it is also the one Optimization Review opens on. One-Time Runs is a filter within it.
What you get there: the individual jobs arriving this way, with their cost and their compute — which is where the all-purpose-compute pattern described above becomes visible per job rather than as a total.
Two things to expect. One-time runs are not exclusively ADF-created, so the filtered list can be broader than the tile's count. And One-Time Runs appears on the SQL Warehouses tab as well, where it describes warehouse activity rather than orchestrated jobs — same words, different question.
What is included in the analysis
Vendor connections, cloud storage reads and Hive/DBFS usage are read from Spark execution telemetry on classic compute in monitored workspaces. Two things follow:
Serverless workloads are not included. A serverless job can read a Hive table, reach an external platform or read from cloud storage exactly like any other job, and none of it appears in these three counts.
Connections that do not go through Spark are not included. A platform reached from driver code — a Python connector, a direct database connection, a REST call — bypasses the mechanism entirely.
SQL warehouses are not included, and a query routed through Lakehouse Federation reads as an internal source rather than as the external system behind it.
External orchestrators works differently. It identifies externally triggered work from the run type and job naming rather than from Spark telemetry, so the two limits above do not describe it. Its own limit is scope: Azure Data Factory is detected today; other schedulers are not yet.
Treat these numbers as a floor rather than a complete picture. They describe what is visible in the monitored, Spark-based part of your estate — which is usually where the largest structural costs sit, but not the whole of it. If a significant share of your workloads runs on serverless compute, talk to us about what coverage would look like for your environment.
📌 Applies to Lakehouse Optimizer 3.5. This guide describes the Orchestration tab as it behaves in version 3.5. Detection coverage and capabilities are extended regularly, so a later version may include opportunity types, filters or destinations that are not covered here, and some of the details described below may have changed. If what you see in the product does not match this page, check your LHO version first — and let us know, so we can bring the page up to date.
Have questions or need help getting started? Please Contact Us for further discussion. We are here to help you make the most out of your Lakehouse Optimizer experience!