Databricks

A claims-triage model on Databricks model serving

Cloud cost at list price, booked by tag, then brought to the books, for September

($6,400 + $1,100) × 0.82

$6,150 booked to the claims initiative for September, at what your invoice says rather than list

The factor is Databricks' bills in your books divided by its list-price usage over the last twelve months. Roiva only measures it: nothing changes until an Owner or Admin accepts it, and a locked month is never touched.

Illustrative figures. Roiva keeps every input with the entry it calculates, and none of it counts until it's approved.

Reads: Monthly cost by product, at list price, from Databricks' system.billing tables, for the last 12 months. Never who ran a job or what it was called. Your own tables only through the metric queries you write, and from those only the period, value, entity and sample_size columns, one row per month.

Keeps: Records of monthly cost by product, plus the totals worked out from them. For each metric query you write, every version of its text and the rows each version returned: a month, a number, a record count, and a breakdown's value, which is whatever the query selects as its entity. These stay after the connection is deleted, because the figures computed from them point to them.

What Roiva keeps from each platform, field by field →

You connect Databricks from inside Roiva, so you need a Roiva workspace first. Sign in to connect it now. New to Roiva? Start free with your work email, take the free assessment and create your workspace at the end, or request an invite.

1. In Databricks: As a workspace admin, open Settings, then Identity and access, then Service principals, and add one for Roiva. On its Secrets tab, generate an OAuth secret with the longest lifetime Databricks allows (730 days). Under Scope, choose Other APIs and add the sql scope only: Roiva asks for a token for SQL and nothing else, and Databricks won't issue one to a secret without it. Copy the client ID and the secret, which Databricks shows only once. Note when it expires: the connection stops syncing then, until a new secret is pasted in
2. In Databricks: Give the service principal Databricks SQL access on its Configurations tab, and Can use on the SQL warehouse Roiva should run on, under that warehouse's Permissions (skip this if All workspace users already has Can use there). A small serverless warehouse is enough
3. In Databricks: As a metastore admin, run the setup script below in the SQL editor, with the service principal's client ID in place of <client-id>. It lets the service principal read the billing tables and nothing else. Unity Catalog must be on for the workspace. If Databricks answers that you don't have MANAGE on Catalog 'system', you aren't a metastore admin: an account admin makes you one in the account console, under Catalog, then the metastore, then Configuration
4. Back in Roiva: Enter the Workspace URL, the address you use for Databricks, and the SQL Warehouse ID from the warehouse's Connection details. Paste the service principal's credentials into API Key / Secret as client_id:client_secret, and save the connection
5. Back in Roiva: The connection page checks each step as the service principal and gives the SQL for anything still missing. Once it's all ticked, click Test Connection
6. Back in Roiva: To count one initiative's spend, give its serving endpoints, jobs, clusters or warehouses a custom tag (roiva-initiative=invoice-ai, say), and enter it under Tag key and Tag value where it's linked to the initiative. The connection page checks each tag too
7. In Databricks: To feed metrics from your own tables, create a view for each in the schema the script makes (workspace.roiva_metrics, or the catalog you chose), which the service principal may read and nothing else. Then in Roiva, set Metrics Schema to that catalog.schema in this connection's settings, and add a query for each metric on its Metric Queries tab

-- Run in the Databricks SQL editor as a metastore admin, with the service
-- principal's client ID (its application ID) in place of <client-id>.
GRANT USE CATALOG ON CATALOG system TO `<client-id>`;
GRANT USE SCHEMA ON SCHEMA system.billing TO `<client-id>`;
GRANT SELECT ON SCHEMA system.billing TO `<client-id>`;

-- For metric queries: a schema of views the service principal may read
-- and nothing else. On a SQL warehouse a view reads its tables with its
-- owner's rights, so the service principal needs no grant on them.
-- A new workspace's catalog is named workspace; an older one's is
-- often main. Use yours in place of workspace.
CREATE SCHEMA IF NOT EXISTS workspace.roiva_metrics;
GRANT USE CATALOG ON CATALOG workspace TO `<client-id>`;
GRANT USE SCHEMA ON SCHEMA workspace.roiva_metrics TO `<client-id>`;
GRANT SELECT ON SCHEMA workspace.roiva_metrics TO `<client-id>`;

Email hello@roiva.ai with what you're seeing, and you'll hear back within one business day.